https://www.sternwarte.uni-erlangen.de/wiki/api.php?action=feedcontributions&feedformat=atom&user=WeberRemeis-Wiki - User contributions [en]2026-05-18T04:58:19ZUser contributionsMediaWiki 1.35.7https://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=4022New Network2026-03-06T13:40:27Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* <s>Check whether the router is set to switch on after power failure in the BIOS</s><br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
* Set IP address of 10G switch via DHCP (doesn't work statically)<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Adjust DHC relay direction<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== 10G Interfaces ====<br />
There is a PCIe card with dual 10G interfaces installed, a Mellanox ConnectX-3. It has two 10G SFP+ ports. OPNsense, which is based on FreeBSD, by default does not load the driver for this card, so it has to be told to load it on boot. This can be done in the boot settings as described in the [https://www.thomas-krenn.com/de/wiki/OPNsense_Chelsio_Mellanox_Broadcom_Netzwerkkarten-Treiber_aktivieren Thomas Krenn Wiki].<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== MDI ===<br />
Ethernet on RJ45 ports consists of a pair of transmitting and receiving wires on both ends. Obviously the receiving pair on one end must be attached to the transmitting pair on the other end. In the old days it was important to consider this property when connecting devices via RJ45 twisted pairs, and in the appropriate case a [https://en.wikipedia.org/wiki/Ethernet_crossover_cable crossover cable] needed to be used. Nowadays ethernet iterfaces can automatically make sure to select the correct twisted pairs to transmit and receive data. This mechanism is called [https://en.wikipedia.org/wiki/Medium-dependent_interface#Automatic_MDI-X automatic MDI-X (automdix)]. It should be enabled on all ports of all switches, and probably is so by default.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB, webgui (defunct) ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB, webgui ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== Ubiquity switch administration ===<br />
The Ubiquiti switches can not really be configured directly. They can only have an "inform url", which is a http address where they pull their configuration from. At this address a server is running which is used to centrally administrate all the switches, called UniFi OS Server. At the time of writing this software is running on libra, reachable via https://libra.sternwarte.uni-erlangen.de:11443/. The password is the current ldap password and the current admin passwort, directly after another.<br />
<br />
=== HP switch administration ===<br />
==== Accessing via CLI ====<br />
Accessing the switches via browser is problematic for older switches since they use stuff like Java in the browser or flash. The CLI always works, if basic connectivity is ensured.<br />
<br />
===== telnet =====<br />
If the switch is up, ethernet is connected, IP address is assigned, subnets are configured correctly, and a computer is in the same VLAN (which should be the case), <code>telnet</code> can be used to access the switches. Just do <code>telnet {IP of switch}</code> and the you should reach the login of the CLI.<br />
<br />
===== serial =====<br />
Alternatively, and especially if the ethernet connection itself doesn't work (yet), the switches can be accessed directly by plugging a computer into the management port(s). Usually those are some kind of serial connection. Older models have a [https://en.wikipedia.org/wiki/D-subminiature D-Sub 9] connector for this purpose, newer models use a RJ45 port. In the latter case RJ45 is only the physical connector. It does not carry ethernet, only a [https://en.wikipedia.org/wiki/RS-232 RS-232] serial connection. Unless the computer itself has a serial port (unlikely these days), in both cases a serial to USB converter is required. Newer switches offer a USB port which internally has a USB to serial converter integrated and shows up in the connecting computer as a serial port. Regardless of the route choosen, if a serial to USB converter is attached the kernel log of the connecting computer should look something like this:<br />
<pre><br />
[ +0,042064] usbcore: registered new interface driver usbserial_generic<br />
[ +0,000009] usbserial: USB Serial support registered for generic<br />
[ +0,001588] usbcore: registered new interface driver ftdi_sio<br />
[ +0,000008] usbserial: USB Serial support registered for FTDI USB Serial Device<br />
[ +0,000101] ftdi_sio 1-4:1.0: FTDI USB Serial Device converter detected<br />
[ +0,000023] usb 1-4: Detected FT232RL<br />
[ +0,006308] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0<br />
</pre><br />
This means the serial port is available under <code>/dev/ttyUSB0</code>. Of course ne number in the end can change, depending on how many serial ports are present. Some devices show up as <code>/dev/ttyACM0</code> instead. These ports can be used with <code>screen</code> to talk to the switch, for example <code>screen /dev/ttyUSB0</code>. If everything goes well a message like <code>Connected with baud rate 9600</code> should show up.<br />
<br />
===== Authentication =====<br />
Depending on the model a username must be entered to access the switch. The "root" user for HP switches is usually called "manager". A password is required to log in in a mode which allows configuration changes. It should be set to our current LDAP admin password. If the password is not or incorrectly entered on the serial console the switches usually drops back to an unpriviledged user which can only look around.<br />
<br />
===== Entering configuration =====<br />
Before anything can be changed in the switch the configuration submenu must be entered. This can be done using the <code>config</code> command:<br />
<pre><br />
switch1# config<br />
switch1(config)# <br />
</pre><br />
Tab completion should work in all circumstances and is often very informative. An ad-hoc set of suggestions and help messages can also be triggered by putting a "?" in the terminal. Note that no "enter" is required in this case. As soon as the "?" is sent the terminal shows the suggestions.<br />
<br />
===== Saving the configuration =====<br />
Note that when changing the configuration of a switch only the running configuration is modified. Changes are not persistent over a reboot of the device. To make the changes persistent the configuration needs to be written to the flash memory of the device. This can be done by <code>write memory</code>.<br />
<br />
Interestingly, <code>write terminal</code> shows all commands necessary to achieve the currently running configuration of the switch.<br />
<br />
===== Basic setup =====<br />
The basic switch setup can be accessed even from outside the <code>config</code> menu using the <code>setup</code> command. It's an interactive menu which is rather intuitive. Among other things, it can be used to set<br />
* The hostname of the switch<br />
* The password for the <code>manager</code> user<br />
* The default gateway<br />
* The IP of the switch in the [[#VLAN|default VLAN]]<br />
<br />
===== Port information =====<br />
Information about the interfaces of the switch can be retrieven by the <code>show interfaces</code> command which lists all interfaces and their information in a table. It can be followed by another option to determine the type of information in the table. These are the options:<br />
* (no option): Sent/received traffic, errors, drops, [https://en.wikipedia.org/wiki/Ethernet_flow_control flow control], broadcast limit<br />
* <code>brief</code> Port type, intrusion alert, enabled/disabled, port status (up/down), mode (speed, duplex), flow control, broadcast limit<br />
* <code>config</code> Port type, enabled/disabled, mode, flow control, [https://en.wikipedia.org/wiki/Medium-dependent_interface MDI] mode<br />
* <code>port-utilization</code> mode (speed, duplex), Rx data, Tx data<br />
Much more detailed ethernet information about specific ports can be obtained by <code>show interfaces ethernet {PORTS}</code>. The ports can be referred to as single ports, a separated list by commas, or ranges separated by "-", or combinations of all. <code>all</code> can be used to refer to all ports at once.<br />
<br />
===== VLAN =====<br />
Information about the configure VLANs can be obtained by the <code>show vlans</code> command. In our case this should only list the default VLAN. VLANs can be configure by entering the context of a specific VLAN, for example <code>vlan 10</code>. In this submenu ports can be assigned to this VLAN as untagged access ports by issuing <code>untagged {PORTS}</code>.<br />
<br />
===== Spanning tree =====<br />
STP can be enabled from the config menu via <code>spanning-tree enable</code>. RSTP can be enforced as the protocol to use via <code>spanning-tree force-version rstp-operation</code>. Port based information about the established tree can be retrieved via <code>show spanning-tree</code>. This gives information about the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Path_cost path cost], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Configuration port priority], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states port state], and designated bridge.<br />
<br />
===== Time =====<br />
The timezones in the HP world are odd. They are given in offsets in minutes. So the timezone of the observatory is +60. It can be set via <code>time timezone 60</code>. The switches can also accept different daylight saving rules. The correct one can be set via <code>time daylight-time-rule Middle-Europe-and-Portugal</code>. The switches are configured to retrieve their time via SNTP, by setting <code>timesync sntp</code>. It utilizes the normal [https://en.wikipedia.org/wiki/Network_Time_Protocol NTP] but has a much simpler implementation on the client side. The switches are configured to use the RRZE NTP servers directly via their IP. This is discouraged, but since the switches don't do DNS not otherwise possible. Information about the current SNTP settings can be retrieved via <code>show sntp</code>. SNTP servers can be set via <code>sntp server {IP}</code> on older models, and <code>sntp server priority 1 {IP}</code> on newer models. The current time can be displayed via <code>time</code>.<br />
<br />
===== MDI Mode =====<br />
The MDI mode can be set via <code>interface {PORT} mdix-mode {MODE}</code> where PORT is a list of ports and MODE is <code>mdi</code>, <code>mdix</code>, or <code>automdix</code>. Obviously we want the latter on all interfaces. To set all ports to auto MDI-X: <code>interface all mdix-mode automdix</code><br />
<br />
==== Accessing via a browser ====<br />
Some of the switches come with an integrated webserver which can be used to access information and perform configuration. While this seems convenient at first, in reality these servers rely on stuff like [https://en.wikipedia.org/wiki/Java_(programming_language) Java] or sometimes even [https://en.wikipedia.org/wiki/Adobe_Flash Adobe Flash] in the browser which is discontinued and simply does not work at all. Therefore using the CLI is the more reliable way.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=4002Meggie Cluster2026-01-20T14:19:25Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== Hardware ==<br />
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow<br />
* <s>Make a list of UEFI settings</s><br />
* <s>Prepare LUNs</s><br />
** <s>Use ZFS zvols as storage backend</s><br />
** <s>Find out which backend type is the best (file, block, SCSI passthrough)</s><br />
** <s>Create a separate dataset for easier zfs setting propagation</s><br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** <s>Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4</s><br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
* Reinstall to Ubuntu 24.0<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
** Ca. 250W per node, 1kW per chassis<br />
* Change batteries in all nodes<br />
* Intel Hyper Threading often goes back to Enabled (double check that it is Disabled) <br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code><br />
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.<br />
The root filesystem size for each node is 80.8GB.<br />
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
<br />
Enter the BIOS menu by pressing F2.<br />
<br />
In the settings, first RESET everything to default.<br />
Then those changes should be made:<br />
<br />
<source><br />
<br />
Main<br />
--> Quiet Boot: Disabled<br />
Advanced<br />
--> Processor Configuration<br />
--> Intel Hyper Threading: Disabled<br />
--> Power & Performance<br />
--> CPU Power & Performance: Performance<br />
--> CPU HWPM State Control<br />
--> Enable CPU HWPM: HWPM Native Mode<br />
--> PCI Configuration<br />
--> NIC Configuration<br />
--> NIC Port 2: Disabled<br />
Server Management<br />
--> Clear System Event Log: Clear it here<br />
Advanced Boot Options<br />
--> System Boot Timeout: 10<br />
--> Bood Mode: UEFI<br />
--> Boot Option Retry: Enabled<br />
<br />
<br />
</source><br />
<br />
REBOOT the system (with e.g. ALT + CMD + DEL)<br />
Then change again in the BIOS settings:<br />
<br />
Intel Hyper Threading often goes back to Enabled (double check that it is Disabled)<br />
<br />
<source><br />
Advanced<br />
--> PCI Configuration<br />
--> UEFI Network Stack<br />
-->IPv6 PXE Support: Disabled<br />
--> UEFI Option ROM Control<br />
--> IPv4 Network Configuration<br />
--> Configured: [x]<br />
--> Enable DHCP: [x]<br />
--> iSCSI Configuration<br />
--> iSCSI Initiator Name: Format is: iqn.1886.de.sternwarte.iscsi:meggie-x-y <br />
where x: rack number (01 is top, 20 is the lowest rack); <br />
y: |-----|-----| for one rack are 4 nodes.<br />
| 1 | 2 |<br />
|-----|-----|<br />
| 3 | 4 |<br />
|-----|-----|<br />
eg: iqn.1886.de.sternwarte.iscsi:meggie-11-1 for node in rack 11 and place 1<br />
--> Add an Attempt<br />
--> MAC-address<br />
--> iSCSI Mode: Enabled<br />
--> Connection Retry Count: 2<br />
--> Connection Establishing Timeout: 5000<br />
--> Enable DHCP: [x]<br />
--> Get target info via DHCP: [x]<br />
--> Authentication Type: None<br />
</source><br />
<br />
<br />
REBOOT and press F6 to enter the boot menu:<br />
<br />
Go to IP based booting option (should be the last of the three) <br />
Select partition:<br />
<source><br />
Ubuntu 20.4 puppet iSCSI autopartition --DO NOT USE IF NO IDEA--<br />
</source><br />
<br />
Be careful, because there exist also other versions like blank.</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=4001Meggie Cluster2026-01-20T14:18:50Z<p>Weber: /* Hardware setup */</p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== Hardware ==<br />
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow<br />
* <s>Make a list of UEFI settings</s><br />
* <s>Prepare LUNs</s><br />
** <s>Use ZFS zvols as storage backend</s><br />
** <s>Find out which backend type is the best (file, block, SCSI passthrough)</s><br />
** <s>Create a separate dataset for easier zfs setting propagation</s><br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** <s>Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4</s><br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
* Reinstall to Ubuntu 24.0<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
** Ca. 250W per node, 1kW per chassis<br />
* Change batteries in all nodes<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code><br />
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.<br />
The root filesystem size for each node is 80.8GB.<br />
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
<br />
Enter the BIOS menu by pressing F2.<br />
<br />
In the settings, first RESET everything to default.<br />
Then those changes should be made:<br />
<br />
<source><br />
<br />
Main<br />
--> Quiet Boot: Disabled<br />
Advanced<br />
--> Processor Configuration<br />
--> Intel Hyper Threading: Disabled<br />
--> Power & Performance<br />
--> CPU Power & Performance: Performance<br />
--> CPU HWPM State Control<br />
--> Enable CPU HWPM: HWPM Native Mode<br />
--> PCI Configuration<br />
--> NIC Configuration<br />
--> NIC Port 2: Disabled<br />
Server Management<br />
--> Clear System Event Log: Clear it here<br />
Advanced Boot Options<br />
--> System Boot Timeout: 10<br />
--> Bood Mode: UEFI<br />
--> Boot Option Retry: Enabled<br />
<br />
<br />
</source><br />
<br />
REBOOT the system (with e.g. ALT + CMD + DEL)<br />
Then change again in the BIOS settings:<br />
<br />
Intel Hyper Threading often goes back to Enabled (double check that it is Disabled)<br />
<br />
<source><br />
Advanced<br />
--> PCI Configuration<br />
--> UEFI Network Stack<br />
-->IPv6 PXE Support: Disabled<br />
--> UEFI Option ROM Control<br />
--> IPv4 Network Configuration<br />
--> Configured: [x]<br />
--> Enable DHCP: [x]<br />
--> iSCSI Configuration<br />
--> iSCSI Initiator Name: Format is: iqn.1886.de.sternwarte.iscsi:meggie-x-y <br />
where x: rack number (01 is top, 20 is the lowest rack); <br />
y: |-----|-----| for one rack are 4 nodes.<br />
| 1 | 2 |<br />
|-----|-----|<br />
| 3 | 4 |<br />
|-----|-----|<br />
eg: iqn.1886.de.sternwarte.iscsi:meggie-11-1 for node in rack 11 and place 1<br />
--> Add an Attempt<br />
--> MAC-address<br />
--> iSCSI Mode: Enabled<br />
--> Connection Retry Count: 2<br />
--> Connection Establishing Timeout: 5000<br />
--> Enable DHCP: [x]<br />
--> Get target info via DHCP: [x]<br />
--> Authentication Type: None<br />
</source><br />
<br />
<br />
REBOOT and press F6 to enter the boot menu:<br />
<br />
Go to IP based booting option (should be the last of the three) <br />
Select partition:<br />
<source><br />
Ubuntu 20.4 puppet iSCSI autopartition --DO NOT USE IF NO IDEA--<br />
</source><br />
<br />
Be careful, because there exist also other versions like blank.</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3765New Network2025-06-26T14:46:16Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* <s>Check whether the router is set to switch on after power failure in the BIOS</s><br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
* Set IP address of 10G switch via DHCP (doesn't work statically)<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Adjust DHC relay direction<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== 10G Interfaces ====<br />
There is a PCIe card with dual 10G interfaces installed, a Mellanox ConnectX-3. It has two 10G SFP+ ports. OPNsense, which is based on FreeBSD, by default does not load the driver for this card, so it has to be told to load it on boot. This can be done in the boot settings as described in the [https://www.thomas-krenn.com/de/wiki/OPNsense_Chelsio_Mellanox_Broadcom_Netzwerkkarten-Treiber_aktivieren Thomas Krenn Wiki].<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== MDI ===<br />
Ethernet on RJ45 ports consists of a pair of transmitting and receiving wires on both ends. Obviously the receiving pair on one end must be attached to the transmitting pair on the other end. In the old days it was important to consider this property when connecting devices via RJ45 twisted pairs, and in the appropriate case a [https://en.wikipedia.org/wiki/Ethernet_crossover_cable crossover cable] needed to be used. Nowadays ethernet iterfaces can automatically make sure to select the correct twisted pairs to transmit and receive data. This mechanism is called [https://en.wikipedia.org/wiki/Medium-dependent_interface#Automatic_MDI-X automatic MDI-X (automdix)]. It should be enabled on all ports of all switches, and probably is so by default.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB, webgui (defunct) ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB, webgui ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== HP switch administration ===<br />
==== Accessing via CLI ====<br />
Accessing the switches via browser is problematic for older switches since they use stuff like Java in the browser or flash. The CLI always works, if basic connectivity is ensured.<br />
<br />
===== telnet =====<br />
If the switch is up, ethernet is connected, IP address is assigned, subnets are configured correctly, and a computer is in the same VLAN (which should be the case), <code>telnet</code> can be used to access the switches. Just do <code>telnet {IP of switch}</code> and the you should reach the login of the CLI.<br />
<br />
===== serial =====<br />
Alternatively, and especially if the ethernet connection itself doesn't work (yet), the switches can be accessed directly by plugging a computer into the management port(s). Usually those are some kind of serial connection. Older models have a [https://en.wikipedia.org/wiki/D-subminiature D-Sub 9] connector for this purpose, newer models use a RJ45 port. In the latter case RJ45 is only the physical connector. It does not carry ethernet, only a [https://en.wikipedia.org/wiki/RS-232 RS-232] serial connection. Unless the computer itself has a serial port (unlikely these days), in both cases a serial to USB converter is required. Newer switches offer a USB port which internally has a USB to serial converter integrated and shows up in the connecting computer as a serial port. Regardless of the route choosen, if a serial to USB converter is attached the kernel log of the connecting computer should look something like this:<br />
<pre><br />
[ +0,042064] usbcore: registered new interface driver usbserial_generic<br />
[ +0,000009] usbserial: USB Serial support registered for generic<br />
[ +0,001588] usbcore: registered new interface driver ftdi_sio<br />
[ +0,000008] usbserial: USB Serial support registered for FTDI USB Serial Device<br />
[ +0,000101] ftdi_sio 1-4:1.0: FTDI USB Serial Device converter detected<br />
[ +0,000023] usb 1-4: Detected FT232RL<br />
[ +0,006308] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0<br />
</pre><br />
This means the serial port is available under <code>/dev/ttyUSB0</code>. Of course ne number in the end can change, depending on how many serial ports are present. Some devices show up as <code>/dev/ttyACM0</code> instead. These ports can be used with <code>screen</code> to talk to the switch, for example <code>screen /dev/ttyUSB0</code>. If everything goes well a message like <code>Connected with baud rate 9600</code> should show up.<br />
<br />
===== Authentication =====<br />
Depending on the model a username must be entered to access the switch. The "root" user for HP switches is usually called "manager". A password is required to log in in a mode which allows configuration changes. It should be set to our current LDAP admin password. If the password is not or incorrectly entered on the serial console the switches usually drops back to an unpriviledged user which can only look around.<br />
<br />
===== Entering configuration =====<br />
Before anything can be changed in the switch the configuration submenu must be entered. This can be done using the <code>config</code> command:<br />
<pre><br />
switch1# config<br />
switch1(config)# <br />
</pre><br />
Tab completion should work in all circumstances and is often very informative. An ad-hoc set of suggestions and help messages can also be triggered by putting a "?" in the terminal. Note that no "enter" is required in this case. As soon as the "?" is sent the terminal shows the suggestions.<br />
<br />
===== Saving the configuration =====<br />
Note that when changing the configuration of a switch only the running configuration is modified. Changes are not persistent over a reboot of the device. To make the changes persistent the configuration needs to be written to the flash memory of the device. This can be done by <code>write memory</code>.<br />
<br />
Interestingly, <code>write terminal</code> shows all commands necessary to achieve the currently running configuration of the switch.<br />
<br />
===== Basic setup =====<br />
The basic switch setup can be accessed even from outside the <code>config</code> menu using the <code>setup</code> command. It's an interactive menu which is rather intuitive. Among other things, it can be used to set<br />
* The hostname of the switch<br />
* The password for the <code>manager</code> user<br />
* The default gateway<br />
* The IP of the switch in the [[#VLAN|default VLAN]]<br />
<br />
===== Port information =====<br />
Information about the interfaces of the switch can be retrieven by the <code>show interfaces</code> command which lists all interfaces and their information in a table. It can be followed by another option to determine the type of information in the table. These are the options:<br />
* (no option): Sent/received traffic, errors, drops, [https://en.wikipedia.org/wiki/Ethernet_flow_control flow control], broadcast limit<br />
* <code>brief</code> Port type, intrusion alert, enabled/disabled, port status (up/down), mode (speed, duplex), flow control, broadcast limit<br />
* <code>config</code> Port type, enabled/disabled, mode, flow control, [https://en.wikipedia.org/wiki/Medium-dependent_interface MDI] mode<br />
* <code>port-utilization</code> mode (speed, duplex), Rx data, Tx data<br />
Much more detailed ethernet information about specific ports can be obtained by <code>show interfaces ethernet {PORTS}</code>. The ports can be referred to as single ports, a separated list by commas, or ranges separated by "-", or combinations of all. <code>all</code> can be used to refer to all ports at once.<br />
<br />
===== VLAN =====<br />
Information about the configure VLANs can be obtained by the <code>show vlans</code> command. In our case this should only list the default VLAN. VLANs can be configure by entering the context of a specific VLAN, for example <code>vlan 10</code>. In this submenu ports can be assigned to this VLAN as untagged access ports by issuing <code>untagged {PORTS}</code>.<br />
<br />
===== Spanning tree =====<br />
STP can be enabled from the config menu via <code>spanning-tree enable</code>. RSTP can be enforced as the protocol to use via <code>spanning-tree force-version rstp-operation</code>. Port based information about the established tree can be retrieved via <code>show spanning-tree</code>. This gives information about the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Path_cost path cost], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Configuration port priority], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states port state], and designated bridge.<br />
<br />
===== Time =====<br />
The timezones in the HP world are odd. They are given in offsets in minutes. So the timezone of the observatory is +60. It can be set via <code>time timezone 60</code>. The switches can also accept different daylight saving rules. The correct one can be set via <code>time daylight-time-rule Middle-Europe-and-Portugal</code>. The switches are configured to retrieve their time via SNTP, by setting <code>timesync sntp</code>. It utilizes the normal [https://en.wikipedia.org/wiki/Network_Time_Protocol NTP] but has a much simpler implementation on the client side. The switches are configured to use the RRZE NTP servers directly via their IP. This is discouraged, but since the switches don't do DNS not otherwise possible. Information about the current SNTP settings can be retrieved via <code>show sntp</code>. SNTP servers can be set via <code>sntp server {IP}</code> on older models, and <code>sntp server priority 1 {IP}</code> on newer models. The current time can be displayed via <code>time</code>.<br />
<br />
===== MDI Mode =====<br />
The MDI mode can be set via <code>interface {PORT} mdix-mode {MODE}</code> where PORT is a list of ports and MODE is <code>mdi</code>, <code>mdix</code>, or <code>automdix</code>. Obviously we want the latter on all interfaces. To set all ports to auto MDI-X: <code>interface all mdix-mode automdix</code><br />
<br />
==== Accessing via a browser ====<br />
Some of the switches come with an integrated webserver which can be used to access information and perform configuration. While this seems convenient at first, in reality these servers rely on stuff like [https://en.wikipedia.org/wiki/Java_(programming_language) Java] or sometimes even [https://en.wikipedia.org/wiki/Adobe_Flash Adobe Flash] in the browser which is discontinued and simply does not work at all. Therefore using the CLI is the more reliable way.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3756New Network2025-06-03T10:50:17Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* <s>Check whether the router is set to switch on after power failure in the BIOS</s><br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Adjust DHC relay direction<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== 10G Interfaces ====<br />
There is a PCIe card with dual 10G interfaces installed, a Mellanox ConnectX-3. It has two 10G SFP+ ports. OPNsense, which is based on FreeBSD, by default does not load the driver for this card, so it has to be told to load it on boot. This can be done in the boot settings as described in the [https://www.thomas-krenn.com/de/wiki/OPNsense_Chelsio_Mellanox_Broadcom_Netzwerkkarten-Treiber_aktivieren Thomas Krenn Wiki].<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== MDI ===<br />
Ethernet on RJ45 ports consists of a pair of transmitting and receiving wires on both ends. Obviously the receiving pair on one end must be attached to the transmitting pair on the other end. In the old days it was important to consider this property when connecting devices via RJ45 twisted pairs, and in the appropriate case a [https://en.wikipedia.org/wiki/Ethernet_crossover_cable crossover cable] needed to be used. Nowadays ethernet iterfaces can automatically make sure to select the correct twisted pairs to transmit and receive data. This mechanism is called [https://en.wikipedia.org/wiki/Medium-dependent_interface#Automatic_MDI-X automatic MDI-X (automdix)]. It should be enabled on all ports of all switches, and probably is so by default.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB, webgui (defunct) ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB, webgui ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== HP switch administration ===<br />
==== Accessing via CLI ====<br />
Accessing the switches via browser is problematic for older switches since they use stuff like Java in the browser or flash. The CLI always works, if basic connectivity is ensured.<br />
<br />
===== telnet =====<br />
If the switch is up, ethernet is connected, IP address is assigned, subnets are configured correctly, and a computer is in the same VLAN (which should be the case), <code>telnet</code> can be used to access the switches. Just do <code>telnet {IP of switch}</code> and the you should reach the login of the CLI.<br />
<br />
===== serial =====<br />
Alternatively, and especially if the ethernet connection itself doesn't work (yet), the switches can be accessed directly by plugging a computer into the management port(s). Usually those are some kind of serial connection. Older models have a [https://en.wikipedia.org/wiki/D-subminiature D-Sub 9] connector for this purpose, newer models use a RJ45 port. In the latter case RJ45 is only the physical connector. It does not carry ethernet, only a [https://en.wikipedia.org/wiki/RS-232 RS-232] serial connection. Unless the computer itself has a serial port (unlikely these days), in both cases a serial to USB converter is required. Newer switches offer a USB port which internally has a USB to serial converter integrated and shows up in the connecting computer as a serial port. Regardless of the route choosen, if a serial to USB converter is attached the kernel log of the connecting computer should look something like this:<br />
<pre><br />
[ +0,042064] usbcore: registered new interface driver usbserial_generic<br />
[ +0,000009] usbserial: USB Serial support registered for generic<br />
[ +0,001588] usbcore: registered new interface driver ftdi_sio<br />
[ +0,000008] usbserial: USB Serial support registered for FTDI USB Serial Device<br />
[ +0,000101] ftdi_sio 1-4:1.0: FTDI USB Serial Device converter detected<br />
[ +0,000023] usb 1-4: Detected FT232RL<br />
[ +0,006308] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0<br />
</pre><br />
This means the serial port is available under <code>/dev/ttyUSB0</code>. Of course ne number in the end can change, depending on how many serial ports are present. Some devices show up as <code>/dev/ttyACM0</code> instead. These ports can be used with <code>screen</code> to talk to the switch, for example <code>screen /dev/ttyUSB0</code>. If everything goes well a message like <code>Connected with baud rate 9600</code> should show up.<br />
<br />
===== Authentication =====<br />
Depending on the model a username must be entered to access the switch. The "root" user for HP switches is usually called "manager". A password is required to log in in a mode which allows configuration changes. It should be set to our current LDAP admin password. If the password is not or incorrectly entered on the serial console the switches usually drops back to an unpriviledged user which can only look around.<br />
<br />
===== Entering configuration =====<br />
Before anything can be changed in the switch the configuration submenu must be entered. This can be done using the <code>config</code> command:<br />
<pre><br />
switch1# config<br />
switch1(config)# <br />
</pre><br />
Tab completion should work in all circumstances and is often very informative. An ad-hoc set of suggestions and help messages can also be triggered by putting a "?" in the terminal. Note that no "enter" is required in this case. As soon as the "?" is sent the terminal shows the suggestions.<br />
<br />
===== Saving the configuration =====<br />
Note that when changing the configuration of a switch only the running configuration is modified. Changes are not persistent over a reboot of the device. To make the changes persistent the configuration needs to be written to the flash memory of the device. This can be done by <code>write memory</code>.<br />
<br />
Interestingly, <code>write terminal</code> shows all commands necessary to achieve the currently running configuration of the switch.<br />
<br />
===== Basic setup =====<br />
The basic switch setup can be accessed even from outside the <code>config</code> menu using the <code>setup</code> command. It's an interactive menu which is rather intuitive. Among other things, it can be used to set<br />
* The hostname of the switch<br />
* The password for the <code>manager</code> user<br />
* The default gateway<br />
* The IP of the switch in the [[#VLAN|default VLAN]]<br />
<br />
===== Port information =====<br />
Information about the interfaces of the switch can be retrieven by the <code>show interfaces</code> command which lists all interfaces and their information in a table. It can be followed by another option to determine the type of information in the table. These are the options:<br />
* (no option): Sent/received traffic, errors, drops, [https://en.wikipedia.org/wiki/Ethernet_flow_control flow control], broadcast limit<br />
* <code>brief</code> Port type, intrusion alert, enabled/disabled, port status (up/down), mode (speed, duplex), flow control, broadcast limit<br />
* <code>config</code> Port type, enabled/disabled, mode, flow control, [https://en.wikipedia.org/wiki/Medium-dependent_interface MDI] mode<br />
* <code>port-utilization</code> mode (speed, duplex), Rx data, Tx data<br />
Much more detailed ethernet information about specific ports can be obtained by <code>show interfaces ethernet {PORTS}</code>. The ports can be referred to as single ports, a separated list by commas, or ranges separated by "-", or combinations of all. <code>all</code> can be used to refer to all ports at once.<br />
<br />
===== VLAN =====<br />
Information about the configure VLANs can be obtained by the <code>show vlans</code> command. In our case this should only list the default VLAN. VLANs can be configure by entering the context of a specific VLAN, for example <code>vlan 10</code>. In this submenu ports can be assigned to this VLAN as untagged access ports by issuing <code>untagged {PORTS}</code>.<br />
<br />
===== Spanning tree =====<br />
STP can be enabled from the config menu via <code>spanning-tree enable</code>. RSTP can be enforced as the protocol to use via <code>spanning-tree force-version rstp-operation</code>. Port based information about the established tree can be retrieved via <code>show spanning-tree</code>. This gives information about the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Path_cost path cost], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Configuration port priority], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states port state], and designated bridge.<br />
<br />
===== Time =====<br />
The timezones in the HP world are odd. They are given in offsets in minutes. So the timezone of the observatory is +60. It can be set via <code>time timezone 60</code>. The switches can also accept different daylight saving rules. The correct one can be set via <code>time daylight-time-rule Middle-Europe-and-Portugal</code>. The switches are configured to retrieve their time via SNTP, by setting <code>timesync sntp</code>. It utilizes the normal [https://en.wikipedia.org/wiki/Network_Time_Protocol NTP] but has a much simpler implementation on the client side. The switches are configured to use the RRZE NTP servers directly via their IP. This is discouraged, but since the switches don't do DNS not otherwise possible. Information about the current SNTP settings can be retrieved via <code>show sntp</code>. SNTP servers can be set via <code>sntp server {IP}</code> on older models, and <code>sntp server priority 1 {IP}</code> on newer models. The current time can be displayed via <code>time</code>.<br />
<br />
===== MDI Mode =====<br />
The MDI mode can be set via <code>interface {PORT} mdix-mode {MODE}</code> where PORT is a list of ports and MODE is <code>mdi</code>, <code>mdix</code>, or <code>automdix</code>. Obviously we want the latter on all interfaces. To set all ports to auto MDI-X: <code>interface all mdix-mode automdix</code><br />
<br />
==== Accessing via a browser ====<br />
Some of the switches come with an integrated webserver which can be used to access information and perform configuration. While this seems convenient at first, in reality these servers rely on stuff like [https://en.wikipedia.org/wiki/Java_(programming_language) Java] or sometimes even [https://en.wikipedia.org/wiki/Adobe_Flash Adobe Flash] in the browser which is discontinued and simply does not work at all. Therefore using the CLI is the more reliable way.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Ubuntu_24&diff=3730Ubuntu 242025-05-13T08:18:42Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
Collecting things to consider/do for the upgrade Ubuntu 24.<br />
<br />
Distinction: System servers and "normal" computers.<br />
<br />
<br />
== Plan ==<br />
<br />
# Block<br />
## Get new central switch<br />
## 10G for router<br />
## Puppet server movement to cygnus<br />
## Test: Configure dnsmasq and ldap on any Ubuntu 24 system (VM)<br />
## Move /data somewhere else<br />
## Get puppet snap management working<br />
# Block<br />
## Setup Ubuntu 24 on vela<br />
## Setup dnsmasq and ldap to vela<br />
## Move puppet server back to vela<br />
# Block<br />
## Setup Ubuntu 24 on libra<br />
## Setup dnsmasq and ldap on libra<br />
# Block<br />
## All other computers<br />
## Slurm<br />
<br />
== Open points ==<br />
* Manage snaps via puppet</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3726New Network2025-05-09T10:41:11Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* <s>Check whether the router is set to switch on after power failure in the BIOS</s><br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Adjust DHC relay direction<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== MDI ===<br />
Ethernet on RJ45 ports consists of a pair of transmitting and receiving wires on both ends. Obviously the receiving pair on one end must be attached to the transmitting pair on the other end. In the old days it was important to consider this property when connecting devices via RJ45 twisted pairs, and in the appropriate case a [https://en.wikipedia.org/wiki/Ethernet_crossover_cable crossover cable] needed to be used. Nowadays ethernet iterfaces can automatically make sure to select the correct twisted pairs to transmit and receive data. This mechanism is called [https://en.wikipedia.org/wiki/Medium-dependent_interface#Automatic_MDI-X automatic MDI-X (automdix)]. It should be enabled on all ports of all switches, and probably is so by default.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB, webgui (defunct) ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB, webgui ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== HP switch administration ===<br />
==== Accessing via CLI ====<br />
Accessing the switches via browser is problematic for older switches since they use stuff like Java in the browser or flash. The CLI always works, if basic connectivity is ensured.<br />
<br />
===== telnet =====<br />
If the switch is up, ethernet is connected, IP address is assigned, subnets are configured correctly, and a computer is in the same VLAN (which should be the case), <code>telnet</code> can be used to access the switches. Just do <code>telnet {IP of switch}</code> and the you should reach the login of the CLI.<br />
<br />
===== serial =====<br />
Alternatively, and especially if the ethernet connection itself doesn't work (yet), the switches can be accessed directly by plugging a computer into the management port(s). Usually those are some kind of serial connection. Older models have a [https://en.wikipedia.org/wiki/D-subminiature D-Sub 9] connector for this purpose, newer models use a RJ45 port. In the latter case RJ45 is only the physical connector. It does not carry ethernet, only a [https://en.wikipedia.org/wiki/RS-232 RS-232] serial connection. Unless the computer itself has a serial port (unlikely these days), in both cases a serial to USB converter is required. Newer switches offer a USB port which internally has a USB to serial converter integrated and shows up in the connecting computer as a serial port. Regardless of the route choosen, if a serial to USB converter is attached the kernel log of the connecting computer should look something like this:<br />
<pre><br />
[ +0,042064] usbcore: registered new interface driver usbserial_generic<br />
[ +0,000009] usbserial: USB Serial support registered for generic<br />
[ +0,001588] usbcore: registered new interface driver ftdi_sio<br />
[ +0,000008] usbserial: USB Serial support registered for FTDI USB Serial Device<br />
[ +0,000101] ftdi_sio 1-4:1.0: FTDI USB Serial Device converter detected<br />
[ +0,000023] usb 1-4: Detected FT232RL<br />
[ +0,006308] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0<br />
</pre><br />
This means the serial port is available under <code>/dev/ttyUSB0</code>. Of course ne number in the end can change, depending on how many serial ports are present. Some devices show up as <code>/dev/ttyACM0</code> instead. These ports can be used with <code>screen</code> to talk to the switch, for example <code>screen /dev/ttyUSB0</code>. If everything goes well a message like <code>Connected with baud rate 9600</code> should show up.<br />
<br />
===== Authentication =====<br />
Depending on the model a username must be entered to access the switch. The "root" user for HP switches is usually called "manager". A password is required to log in in a mode which allows configuration changes. It should be set to our current LDAP admin password. If the password is not or incorrectly entered on the serial console the switches usually drops back to an unpriviledged user which can only look around.<br />
<br />
===== Entering configuration =====<br />
Before anything can be changed in the switch the configuration submenu must be entered. This can be done using the <code>config</code> command:<br />
<pre><br />
switch1# config<br />
switch1(config)# <br />
</pre><br />
Tab completion should work in all circumstances and is often very informative. An ad-hoc set of suggestions and help messages can also be triggered by putting a "?" in the terminal. Note that no "enter" is required in this case. As soon as the "?" is sent the terminal shows the suggestions.<br />
<br />
===== Saving the configuration =====<br />
Note that when changing the configuration of a switch only the running configuration is modified. Changes are not persistent over a reboot of the device. To make the changes persistent the configuration needs to be written to the flash memory of the device. This can be done by <code>write memory</code>.<br />
<br />
Interestingly, <code>write terminal</code> shows all commands necessary to achieve the currently running configuration of the switch.<br />
<br />
===== Basic setup =====<br />
The basic switch setup can be accessed even from outside the <code>config</code> menu using the <code>setup</code> command. It's an interactive menu which is rather intuitive. Among other things, it can be used to set<br />
* The hostname of the switch<br />
* The password for the <code>manager</code> user<br />
* The default gateway<br />
* The IP of the switch in the [[#VLAN|default VLAN]]<br />
<br />
===== Port information =====<br />
Information about the interfaces of the switch can be retrieven by the <code>show interfaces</code> command which lists all interfaces and their information in a table. It can be followed by another option to determine the type of information in the table. These are the options:<br />
* (no option): Sent/received traffic, errors, drops, [https://en.wikipedia.org/wiki/Ethernet_flow_control flow control], broadcast limit<br />
* <code>brief</code> Port type, intrusion alert, enabled/disabled, port status (up/down), mode (speed, duplex), flow control, broadcast limit<br />
* <code>config</code> Port type, enabled/disabled, mode, flow control, [https://en.wikipedia.org/wiki/Medium-dependent_interface MDI] mode<br />
* <code>port-utilization</code> mode (speed, duplex), Rx data, Tx data<br />
Much more detailed ethernet information about specific ports can be obtained by <code>show interfaces ethernet {PORTS}</code>. The ports can be referred to as single ports, a separated list by commas, or ranges separated by "-", or combinations of all. <code>all</code> can be used to refer to all ports at once.<br />
<br />
===== VLAN =====<br />
Information about the configure VLANs can be obtained by the <code>show vlans</code> command. In our case this should only list the default VLAN. VLANs can be configure by entering the context of a specific VLAN, for example <code>vlan 10</code>. In this submenu ports can be assigned to this VLAN as untagged access ports by issuing <code>untagged {PORTS}</code>.<br />
<br />
===== Spanning tree =====<br />
STP can be enabled from the config menu via <code>spanning-tree enable</code>. RSTP can be enforced as the protocol to use via <code>spanning-tree force-version rstp-operation</code>. Port based information about the established tree can be retrieved via <code>show spanning-tree</code>. This gives information about the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Path_cost path cost], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Configuration port priority], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states port state], and designated bridge.<br />
<br />
===== Time =====<br />
The timezones in the HP world are odd. They are given in offsets in minutes. So the timezone of the observatory is +60. It can be set via <code>time timezone 60</code>. The switches can also accept different daylight saving rules. The correct one can be set via <code>time daylight-time-rule Middle-Europe-and-Portugal</code>. The switches are configured to retrieve their time via SNTP, by setting <code>timesync sntp</code>. It utilizes the normal [https://en.wikipedia.org/wiki/Network_Time_Protocol NTP] but has a much simpler implementation on the client side. The switches are configured to use the RRZE NTP servers directly via their IP. This is discouraged, but since the switches don't do DNS not otherwise possible. Information about the current SNTP settings can be retrieved via <code>show sntp</code>. SNTP servers can be set via <code>sntp server {IP}</code> on older models, and <code>sntp server priority 1 {IP}</code> on newer models. The current time can be displayed via <code>time</code>.<br />
<br />
===== MDI Mode =====<br />
The MDI mode can be set via <code>interface {PORT} mdix-mode {MODE}</code> where PORT is a list of ports and MODE is <code>mdi</code>, <code>mdix</code>, or <code>automdix</code>. Obviously we want the latter on all interfaces. To set all ports to auto MDI-X: <code>interface all mdix-mode automdix</code><br />
<br />
==== Accessing via a browser ====<br />
Some of the switches come with an integrated webserver which can be used to access information and perform configuration. While this seems convenient at first, in reality these servers rely on stuff like [https://en.wikipedia.org/wiki/Java_(programming_language) Java] or sometimes even [https://en.wikipedia.org/wiki/Adobe_Flash Adobe Flash] in the browser which is discontinued and simply does not work at all. Therefore using the CLI is the more reliable way.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Ubuntu_24&diff=3712Ubuntu 242025-05-06T11:47:55Z<p>Weber: Created page with "Category:Admin Collecting things to consider/do for the upgrade Ubuntu 24. Distinction: System servers and "normal" computers. == Plan == # Block ## Get new central s..."</p>
<hr />
<div>[[Category:Admin]]<br />
<br />
Collecting things to consider/do for the upgrade Ubuntu 24.<br />
<br />
Distinction: System servers and "normal" computers.<br />
<br />
<br />
== Plan ==<br />
<br />
# Block<br />
## Get new central switch<br />
## 10G for router<br />
## Puppet server movement to cygnus<br />
## Test: Configure dnsmasq and ldap on any Ubuntu 24 system (VM)<br />
## Move /data somewhere else<br />
# Block<br />
## Setup Ubuntu 24 on vela<br />
## Setup dnsmasq and ldap to vela<br />
## Move puppet server back to vela<br />
# Block<br />
## Setup Ubuntu 24 on libra<br />
## Setup dnsmasq and ldap on libra<br />
# Block<br />
## All other computers<br />
## Slurm<br />
<br />
== Open points ==<br />
* Manage snaps via puppet</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3707Meggie Cluster2025-05-05T13:49:35Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== Hardware ==<br />
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow<br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
** Use ZFS zvols as storage backend<br />
** Find out which backend type is the best (file, block, SCSI passthrough)<br />
** Create a separate dataset for easier zfs setting propagation<br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
** Ca. 250W per node, 1kW per chassis<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Admin&diff=3688Admin2025-04-25T06:23:40Z<p>Weber: </p>
<hr />
<div>===== Remeis Admin-Page =====<br />
<br />
This page is only internally visible to the Admins.<br />
<br />
Things which need to be documented:<br />
* Ansible<br />
* Updates (using unattended-upgrades)<br />
* Puppet<br />
* Storage volumes (raids, filesystems, oddities, etc.)<br />
<br />
==== Lab Course ====<br />
<br />
* Before the lab course: <br />
** check the PCs in the Meridian building:<br />
*** they should all be functional, broken ones should be replaced<br />
*** in total we should have about 10 computers available for the students<br />
** [[setup prakti|Setup the prakti accounts]]<br />
* [[Computer_Introduction|Introduction to Computers/Linux for Lab Students ]]<br />
<br />
==== Serendipitous Issues ====<br />
<br />
* [[Login Problems|Typical login problems]]<br />
* [[Unmet Dependencies|''unmet dependencies'' issue in apt-get]]<br />
* [[Overflow Filesystem|overflow file system]]<br />
<br />
==== Add new Users ====<br />
* Go to the remeis internal homepage -> admin -> LDAP<br />
* Fill out the necessary fields, including a valid email address. If possible, also fill out the fields for room and telephone number.<br />
* Log into LDAP ( https://www.sternwarte.uni-erlangen.de/phpldapadmin/), go to import and copy the text of the previous step. '''Important''': if the user is external, "current" in the first line has to be changed to "external"<br />
* set the password of the user: Use the LDAP query <code>ldappasswd -H ldap://vela -x -D "cn=admin,dc=sternwarte,dc=uni-erlangen,dc=de" -W -S "uid={USERNAME},ou=current,ou=People,dc=sternwarte,dc=uni-erlangen,dc=de"</code> while obviously adjusting the LDAP dn.<br />
* log in as ubuntuadmin on libra and create the home directory like that: <code>sudo mkdir /exports/disk1/{USERNAME} && sudo chown {USERNAME}:remeis /exports/disk1/{USERNAME}</code><br />
* Set the quota (on libra), using a prototype which has normal quota <code>sudo edquota -p [prototype] [username]</code> (normal Quota is 8GB softlimit and 12GB hardlimit; it can be manually edited/shown with edquota -u user)<br />
* Add the default cshrc to /home/username/.cshrc <br />
* If desired (in most cases), create a folder for the user at ''/userdata/data/username'' and adjust the permissions with <code>sudo chown username:remeis /userdata/data/username</code><br />
* Send Email to admin about the new user<br />
<br />
==== Running Software / Updates / Setups ====<br />
<br />
In this section a selection of services and their current configuration is described. <br />
<br />
* [[CUPS| CUPS]] (Printing Service)<br />
* [[Virtual Boxes| Virtual Boxes]]<br />
<br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Admin&diff=3686Admin2025-04-24T12:31:03Z<p>Weber: </p>
<hr />
<div>===== Remeis Admin-Page =====<br />
<br />
This page is only internally visible to the Admins.<br />
<br />
Things which need to be documented:<br />
* Ansible<br />
* Updates (using unattended-upgrades)<br />
* Puppet<br />
<br />
==== Lab Course ====<br />
<br />
* Before the lab course: <br />
** check the PCs in the Meridian building:<br />
*** they should all be functional, broken ones should be replaced<br />
*** in total we should have about 10 computers available for the students<br />
** [[setup prakti|Setup the prakti accounts]]<br />
* [[Computer_Introduction|Introduction to Computers/Linux for Lab Students ]]<br />
<br />
==== Serendipitous Issues ====<br />
<br />
* [[Login Problems|Typical login problems]]<br />
* [[Unmet Dependencies|''unmet dependencies'' issue in apt-get]]<br />
* [[Overflow Filesystem|overflow file system]]<br />
<br />
==== Add new Users ====<br />
* Go to the remeis internal homepage -> admin -> LDAP<br />
* Fill out the necessary fields, including a valid email address. If possible, also fill out the fields for room and telephone number.<br />
* Log into LDAP ( https://www.sternwarte.uni-erlangen.de/phpldapadmin/), go to import and copy the text of the previous step. '''Important''': if the user is external, "current" in the first line has to be changed to "external"<br />
* set the password of the user: Use the LDAP query <code>ldappasswd -H ldap://vela -x -D "cn=admin,dc=sternwarte,dc=uni-erlangen,dc=de" -W -S "uid={USERNAME},ou=current,ou=People,dc=sternwarte,dc=uni-erlangen,dc=de"</code> while obviously adjusting the LDAP dn.<br />
* log in as ubuntuadmin on libra and create the home directory like that: <code>sudo mkdir /exports/disk1/{USERNAME} && sudo chown {USERNAME}:remeis /exports/disk1/{USERNAME}</code><br />
* Set the quota (on libra), using a prototype which has normal quota <code>sudo edquota -p [prototype] [username]</code> (normal Quota is 8GB softlimit and 12GB hardlimit; it can be manually edited/shown with edquota -u user)<br />
* Add the default cshrc to /home/username/.cshrc <br />
* If desired (in most cases), create a folder for the user at ''/userdata/data/username'' and adjust the permissions with <code>sudo chown username:remeis /userdata/data/username</code><br />
* Send Email to admin about the new user<br />
<br />
==== Running Software / Updates / Setups ====<br />
<br />
In this section a selection of services and their current configuration is described. <br />
<br />
* [[CUPS| CUPS]] (Printing Service)<br />
* [[Virtual Boxes| Virtual Boxes]]<br />
<br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3685Meggie Cluster2025-04-24T12:26:10Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow<br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
** Use ZFS zvols as storage backend<br />
** Find out which backend type is the best (file, block, SCSI passthrough)<br />
** Create a separate dataset for easier zfs setting propagation<br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
** Ca. 250W per node, 1kW per chassis<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=List_of_servers_in_the_Remeis_Cluster&diff=3680List of servers in the Remeis Cluster2025-04-14T14:19:25Z<p>Weber: </p>
<hr />
<div>'''This page is very outdated!'''<br />
<br />
<br />
Here you can find an overview of our servers and what pieces of software and services are running on each of them.<br />
<br />
'''crux'''<br />
* LDAP<br />
* Web<br />
* DNS<br />
* DHCP<br />
* NTP<br />
* /data<br />
* git<br />
* Mail<br />
* Backup (von /data)<br />
* Ubuntu-Mirror<br />
* MySQL<br />
<br />
'''leo'''<br />
* /userdata<br />
* 2 Vboxes: auriga: print & hydrus: web<br />
* cronjobs<br />
* torque server + client<br />
* Randomnumber-Server<br />
<br />
'''phoenix'''<br />
* /home<br />
* Backup (von /home)<br />
* /ixo<br />
* cronjobs<br />
* quota<br />
<br />
'''grus'''<br />
* /eu<br />
* cronjobs<br />
<br />
'''draco'''<br />
* /loft<br />
* VBox: openproject<br />
* MySQL<br />
* cronjob<br />
<br />
'''virgo'''<br />
* /virgoabc<br />
<br />
'''serpens'''<br />
* usage only for erosita project<br />
* web (erosita)<br />
<br />
[[Category:Internal]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=How_to_use_the_Remeis_cluster&diff=3679How to use the Remeis cluster2025-04-14T14:17:41Z<p>Weber: </p>
<hr />
<div>== How To use the Remeis Cluster ==<br />
<br />
The Cluster of the Remeis observatory is designed to make all resources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:admin@sternwarte.uni-erlangen.de admin@sternwarte.uni-erlangen.de].<br />
<br />
=== Performing Computations ===<br />
<br />
* You should never log in on servers directly and especially not run any computations on servers without using slurm.<br />
* Computations and extractions on the Cluster should be submitted via '''slurm'''. Information on how to use it can be found [[Slurm|here]].<br />
* Also interactive sessions can be started via '''slurm'''.<br />
<br />
=== The Data-Storage System ===<br />
<br />
* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backed up simply. And there is no quota on /userdata.<br />
<br />
* '''/home:''' This is the place for self-developed software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backed up every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory for the worst case (e.g., fire). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (5 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on libra, like <br />
ssh libra quota -s -f /exports/disk1<br />
<br />
* '''/scratch:''' For temporary or otherwise not very important data. You should not store anything important on /scratch!! The scratch disks are NOT backedup and the scratch disks are not fail safe, i.e. if a scratch disk is broken (which happens regularly), your data is GONE. scratch might be emptied at any time without warning.<br />
<br />
* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to Ingo.<br />
<br />
=== Logging into Remeis ===<br />
<br />
==== Remote Login ====<br />
<br />
===== Using ssh =====<br />
You can log-in via ssh simply by <br />
ssh -X <username>@<host>.sternwarte.uni-erlangen.de<br />
<br />
<br />
===== Using x2go =====<br />
<br />
For remote login with a graphical interface you can use [[X2Go Remote Desktop]]<br />
<br />
===== Hosts =====<br />
Host names are generally the constellation names, e.g., <code>norma</code> or <code>mensa</code>.<br />
<br />
==== Connecting without password and short hostnames ====<br />
<br />
Sick of always typing <code>user@myfavouritehost.sternwarte.uni-erlangen.de</code> and always having to enter your password when switching machines inside inside the Remeis Cluster? The follwing setup will make your life easier.<br />
<br />
===== Config-File =====<br />
<br />
In the file <code>~/.ssh/config</code> you can set up the configuration for ssh. A few things are really useful.<br />
# activate compression, might lead to a faster connection<br />
Compression yes<br />
# do X11 forwarding by default, i.e., the option "-X" is <br />
# always set when using "ssh"<br />
forwardX11 yes<br />
<br />
In order to define a short-cut for a host, you have to add the following information to the ''~/.ssh/config''<br />
# the short-name you want to call the host (most useful is the general name of the machine<br />
Host=norma<br />
# now define user and host such that ''<User>@<Hostname>'' is the correct adress<br />
Hostname=norma.sternwarte.uni-erlangen.de<br />
User=YourUsername<br />
<br />
===== ssh-keys =====<br />
<br />
If you setup the ssh-keys properly, you can login via ssh without entering a password (although you should definitely protect your ssh key with a password). A detailed description can be found [[SSH-Keys|here]].<br />
<br />
=== Setting the Default Options/Software ===<br />
<br />
Each new account is equipped with a sample file at ''~/.cshrc'', which is loaded on every start of the terminal. Hence, this defines your options, i.e., the software available, short-cuts, your favourite editor, and so on. Commonly useful options are activated by default. In addition, there are a few additional features, which are commented out by default but still very useful for most users.<br />
<br />
# set your favourite text editor to be used by default; important, e.g., if you want to use GIT<br />
setenv EDITOR jed <br />
<br />
# activate the X-ray software (HEADAS, satellite extraction scripts, ...)<br />
source /data/software/cshrc.in <br />
<br />
<br />
More information on the TC shell and how to configure it can be found [[TcShell|here]].<br />
<br />
[[Category:Current Members]][[Category:Software]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Login&diff=3678Login2025-04-14T08:13:39Z<p>Weber: </p>
<hr />
<div>== Ubuntu 20.04 Issues ==<br />
<br />
For some reason it can happen that for a particular user on a particular machine lightdm (login manager) decides to load the gnome session (albeit it should load xfce per default). It does so despite no full gnome desktop environment is installed. It seems that somehow the d-bus is wrongly configured (?) or started (?), not sure. Anyway, the effect is that even logout or restart does not fix this. The user is greeted with a pretty blank gnome desktop environment running on top of xfce (the later is therefore inaccessible).<br />
<br />
This seems to fix this:<br />
<code><br />
sudo dbus-send --system --type=method_call --print-reply --dest=org.freedesktop.Accounts /org/freedesktop/Accounts/User<UID> org.freedesktop.Accounts.User.SetXSession string:xfce<br />
</code><br />
make sure to replace <code><UID></code> with the affected users id.<br />
<br />
== Issuses during update to 16.04 ==<br />
<br />
Especially in the course of the update of 14.04 to 16.04 the following issues have turned out to happen quite often. The text was copied from Ingo's mail to the admin list.<br />
<br />
* the .dmrc problem: if something else than "plasma" is specified in<br />
this file in the home directory of the user, you'll get a "can not load<br />
session" (or similar) error message. Since lightdm sort of "caches" the<br />
value of this file in a file of its own, this can be very annoying to<br />
sort out.<br />
<br />
* the .Xauthority problem: can happen if a machine is newly installed<br />
and the user has worked on that machine before the update, i.e. a wrong<br />
key is stored in .Xauthority<br />
<br />
* akonadi / baloo crap: if enabled, these daemons cause heavy IO load<br />
and potentially slow down / inhibit login or make the machine barely<br />
usable while creating the databases<br />
<br />
* useless KDE animations: on older or weaker machines, enabling (or not<br />
disabling) tons of KDE animations can make the machine very slow, or<br />
might even crash the graphics driver<br />
<br />
* reboot necessary: as it turned out in Marias case, the machine needed<br />
a reboot. It seems that the new KDE is new more touchy w.r.t. a<br />
necessary reboot. I.e. if KDE components have been updated by the<br />
nightly cron jobs, the KDE/Ubuntu 16 system services (dbus et al)<br />
<br />
<br />
apparantly do not get along very well any more causing these frozen<br />
logins and a reboot is needed.<br />
==> if this happens more often, we have to think about whether we can<br />
continue with the nightly updates, or whether we do them only once a<br />
week and have reboots afterwards (something like "reboot Monday" or<br />
similar).<br />
<br />
* ''possibly'' some other cron job may interfere with the new KDE causing<br />
these frozen logins.<br />
<br />
<br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=LDAP&diff=3677LDAP2025-04-14T08:13:07Z<p>Weber: </p>
<hr />
<div><br />
<br />
[[Category:Admin]]<br />
= LDAP =<br />
The LDAP server at Remeis is running on vela. The service itself is called ''slapd''. A backup LDAP server is running on libra.<br />
<br />
Every interaction with the ''slapd'' requires root access.<br />
<br />
== Where LDAP is used ==<br />
LDAP is not only used for the authentication to login into the variety of machines at Remeis. Also a lot of other services/scripts use ldap:<br />
<br />
* This wiki<br />
* The internal webpage<br />
* [[GitLab]]<br />
* [[#phpLDAPadmin|phpLDAPadmin]]<br />
* The <code>who_takes_the_minutes.pl</code> script of the X-ray group<br />
* The ''cgnotifier'' for the [[cgroups]]<br />
<br />
== Configuration ==<br />
<br />
=== Config files ===<br />
The main LDAP configuration file is <code>/etc/ldap.conf</code>. Note that this is not the default location for the configuration file. Therefore the defaults for the slapd are changed in <code>/etc/default/slapd</code>, see the entry <code>SLAPD_CONF</code>. The <code>/etc/ldap.conf</code> is also present on all the clients and specifies via IP-Address which LDAP server to contact for authentication. The LDAP requests are then handled on taurus by the ''slapd'' which is configured via <code>/etc/ldap/slapd.conf</code>.<br />
<br />
=== phpLDAPadmin ===<br />
phpLDAPadmin is a web interface based on [https://de.wikipedia.org/wiki/PHP PHP] which allows simple modifications to the directory for example to [[:Category:Admin#Add new Users|add new users]] or change group memberships. It is accessible via the [https://www.sternwarte.uni-erlangen.de/intern/phpldapadmin/ internal webpages] and only out of the network at the Remeis observatory.<br />
<br />
==== Configuration of phpLADPadmin ====<br />
The configuration files for phpLDAPadmin are located in <code>/etc/phpldapadmin/</code> on the webserver, in this case [[cetus]]. The important file is <code>config.php</code> which holds information about which LDAP servers can be queried, in this case [[taurus]].<br />
<br />
<br />
== Starting/Stopping the service ==<br />
The slapd can be stopped via [https://de.wikipedia.org/wiki/Systemd systemd]:<br />
<pre><br />
systemctl stop slapd<br />
</pre><br />
'''Important:''' If you do so no authentication on the whole cluster is possible anymore. You can not open a new shell on any host until you start the service again!<br />
<br />
Starting the ''slapd'':<br />
<pre><br />
systemctl start slapd<br />
</pre><br />
You also can have a look at the output produced by the running service via<br />
<pre><br />
systemctl status slapd<br />
</pre><br />
<br />
== Database ==<br />
The [https://en.wikipedia.org/wiki/Berkeley_DB Berkeley DB] containing all the active directory information is located at <code>/var/lib/ldap</code>. Files in this directory must not be touched manually in any way. The only secure way to see the content of the database is by using ''slapcat''. For example<br />
<pre><br />
slapcat -f /etc/ldap/slapd.conf<br />
</pre><br />
will show all entries of the database.<br />
<br />
== Backup ==<br />
Each night the database is dumped to <code>/data/system/backup/ldap/</code> via a cronjob in <code>/etc/cron.d/ldap_backup</code>. Also a tarball of the ''slapd'' and <code>ldap.conf</code> is created there. All backups older than 14 days are deleted.</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Install_and_manage_software&diff=3676Install and manage software2025-04-14T08:12:11Z<p>Weber: </p>
<hr />
<div>'''This page does not refer to the installation of packages using the package manager of the operating system'''<br />
<br />
The software locally compiled on the cluster is structured within a folder and<br />
provided by a single mount point containing the compiled software, shell<br />
scripts which define the installation procedure and the defined module files<br />
for lmod.<br />
<br />
== Structure of the local software ==<br />
<br />
Lmod allows three principal levels of hierarchy to separate the software a user<br />
can access (we will generally use '''module''' for a piece of software, no<br />
matter if it is a binary, library, script or something else). This hierarchy<br />
allows to hide modules from the user in case certain requirements are not<br />
fulfilled.<br />
<br />
=== Visible but category separated ===<br />
<br />
The most simple hierarchy is by assigning each module to a category. This is<br />
done by the placing the module in a sub-directory of the software mount point.<br />
The category can be any string that is allowed as directory name but should<br />
be concise and meaningful (e.g., Development, Computing, Science, ...).<br />
This categorization does not give any restrictions on the load-ability of the<br />
module and any user can see all modules in this categories.<br />
<br />
=== Not visible until requirements are met ===<br />
<br />
For each module there must be a definition file for lmod written as a lua<br />
script which defines the access to the module. Lmods principle working<br />
scheme is to change the user environment on the fly depending on how it is<br />
defined in the definition file. The definition files are searched in paths<br />
given by environment variables and in this combination modules can change the<br />
search path for modules! Using this one can define modules that are only<br />
visible once certain other modules are loaded. Before that the user has no clue<br />
that these modules exist.<br />
<br />
This sort of hierarchy is very useful when hiding modules that are in a general<br />
way not usable by the user and a user would be very surprised if he/she could<br />
see that a certain module exist but can not be used. For the remeis cluster<br />
we use this hierarchy to separate modules build for different OS systems. In<br />
this way a user can not accidentally load a module that is defined for OS 1 but<br />
is logged in on a machine with OS 2. Further, if there are modules only<br />
available for one OS the user is not confused because the module is in<br />
principle listed but marked as unusable.<br />
<br />
Currently we have no other needs to hide software from the user such that<br />
there are additional directory's in the software mount with the name of the OS<br />
they are working on. In these directories there is again the category structure<br />
as described above.<br />
<br />
=== Extensions to a module ===<br />
<br />
Similar to the above hierarchy is an '''extension''' a mechanism provided by<br />
lmod which allows to mark a module as having extension modules. These modules<br />
are only loadable if the ''base''module (i.e, the one that is to be extended)<br />
is loaded. This is particularly useful if, for example, a binary depends on<br />
other locally compiled software and is strongly version bound to that library<br />
(e.g., isis and heasoft).<br />
<br />
The extensions are in general visible but loading them tells the user that the<br />
modules can only be loaded when other modules are loaded first. This is useful<br />
for modules a user should have access to but need this level of hierarchy such<br />
that no user is confronted with a loadable but not working module.<br />
<br />
While for the other two hierarchies the modules and module definition files have<br />
the same structure of organization (if a module is in a category folder, the<br />
definition file is in a category folder with the same name) the extension<br />
definition files are in a separate ''category'' in the mount point called<br />
'''Extensions''' (again containing category directories) while the modules are<br />
in the category directories in the mount point itself.<br />
<br />
(The similar structure for the modules and the definition files is by no means<br />
necessary but for better maintainability it should be kept whenever possible.)<br />
<br />
== Installation of new modules ==<br />
<br />
We made the experience that often old software is used/required by certain<br />
users and re-installing them required tremendous amount of work. To overcome<br />
this issue the install process for any module should be defined by a bash<br />
script which performs the installation.<br />
<br />
Each module should have its own install script (also for each version a<br />
separate one). The contents of this script are arbitrary but it should take one<br />
argument which is either ''install'' or ''uninstall'' and then do the<br />
appropriate tasks.<br />
<br />
A skeleton for such a script is given in the software mount as skel-0.0.0.sh.<br />
The naming of this file is not arbitrary and allows to check consistency of the<br />
installed modules, the definition files and the scripts that do the<br />
installation. The convention for the file naming is given below. The scripts<br />
handling the installation process together with the definition files (and the<br />
basic directory structure) are version controlled via git in<br />
www.sternwarte.uni-erlangen.de:Admins/remeis-software.git<br />
to have better documentation.<br />
<br />
=== Naming conventions ===<br />
<br />
The modules are installed in a directory that is '''only'' the version of the<br />
module, this directory is place in a directory with '''only''' the module name,<br />
e.g.,<br />
<Category/path>/module-name/version/{bin,lib,...}<br />
The install script should reside in the module-name/ directory and be named as<br />
module-name-version.sh<br />
or<br />
module-name-version+E-extends-version.sh<br />
The <nowiki>+E-...</nowiki> part marks that this script installs a module that<br />
extends the named module with the given version. For examples check the<br />
structure in the git repository.<br />
<br />
Module names should be all lowercase to separate it from the categories (with<br />
first letter in upper case).<br />
<br />
== Writing definition files ==<br />
<br />
The general process of writing definition files should not be discussed here as<br />
the documentation of lmod is very exhaustive (https://lmod.readthedocs.io/en/latest/).<br />
<br />
Only as a general remark: Definition files are written in lua and the naming of<br />
the files must be the modules version (and determines which module is<br />
loaded when). Additionally, there are convenience functions defined in a file<br />
called SitePackage.lua which is respected by lmod (see docs). In case<br />
definition files share more complicated functions it is best to place them in<br />
there.<br />
<br />
== How modules are accessed ==<br />
<br />
Lmod is provided by a set of environment variables which have to be defined in<br />
each users environment. The best way to do this is the global login scripts in<br />
/etc/.... A script that sets all relevant variables is given is profile.in and<br />
cshrc.in in the software mount point.<br />
<br />
The lua interpreter necessary to use the module files is itself installed in<br />
the software mount just like any other module and as such a relocation of the<br />
whole structure should be easy. All module definition files and install scripts<br />
should respect the variable SOFTWARE which must be set to the mount point.<br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Grafana&diff=3675Grafana2025-04-14T08:11:20Z<p>Weber: </p>
<hr />
<div>= What is it for =<br />
Grafana can be used to visualize any kind of data. In this case it is mainly used to visualize monitoring data captured by [[Prometheus]] (and other datasources).<br />
<br />
= Access =<br />
The Grafana server can be accessed via [https://www.sternwarte.uni-erlangen.de/grafana www.sternwarte.uni-erlangen.de/grafana].<br />
<br />
= Usage =<br />
Grafana has a very intuitive graphical interface running directly in the browser. The different kind of data is organized in so called '''dashboards'''. The active dashboard can be changed using the four squares in the upper right-hand corner. In the upper left-hand corner the time range can be adjusted. For further help please consult the official grafana [http://grafana.com website] .<br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=DHCP&diff=3674DHCP2025-04-14T08:10:58Z<p>Weber: </p>
<hr />
<div><br />
<br />
[[Category:Admin]]<br />
= DHCP server =<br />
The main DHCP server at Remeis is running on [[vela]], a secondory one is running on [[libra]].<br />
<br />
= Configuration of the DHCP Server =<br />
[[puppet]] is used to automatically configure the dhcp server for which we chose [https://en.wikipedia.org/wiki/Dnsmasq dnsmasq], thus it also provides the [[DNS]] service.<br />
If you want to change the configuration of the DHCP server, clone the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet repository], do your changes in the config files and templates there <code>modules/remeis/manifest/dnsmasq.pp</code>, <code>modules/remeis/files/dnsmasq/</code>, and <code>modules/remeis/templates/</code> respectively.<br />
When you push, some checking is done on the new configuration and if it is fine puppet will apply it in the defined time interval.</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=CUPS&diff=3673CUPS2025-04-14T08:08:04Z<p>Weber: </p>
<hr />
<div><br />
==== Printing Service CUPS ====<br />
<br />
Printing at the observatory is handled by [https://www.cups.org/index.html CUPS].<br />
CUPS is installed in the virtual box auriga (which lives on taurus) and can be administered through a web interface running on port 631: http://auriga:631/ (accessible only from within the institute).<br />
<br />
Through this web interface, printers can be added or modified, queues can be stopped, jobs canceled, and other other maintenance jobs can be performed. To perform such tasks, CUPS requires you to login: use '''ubuntuadmin''' and the corresponding password.<br />
<br />
When addong new printers, do forget to enable '''sharing''' to make the printer visible in the entire cluster.<br />
<br />
Check out the [https://www.cups.org/documentation.html CUPS documentation] how to work with CUPS.<br />
<br />
=== Trouble shooting ===<br />
<br />
If a printer is not responding/not accessible: <br />
* make sure that the printer is online (also check the network cable!), working properly, and not just busy. <br />
* check the web interface of that printer, try to print a test page from the web interface.<br />
* check the web interface for jobs in that queue. Remove failed jobs.<br />
<br />
If no printers are availble (or just a generic printer), CUPS is probably down:<br />
* login to auriga as ubuntuadmin<br />
* restart the CUPS service:<br />
<pre><br />
sudo systemctl restart cups<br />
sudo systemctl restart cups<br />
</pre><br />
* sometimes it might be necessary to reboot the entire virtual box (CUPS will automatically be started after the reboot):<br />
<pre><br />
sudo shutdown -r now<br />
</pre><br />
<br />
=== List of Printers on the Cluster ===<br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Overflow_Filesystem&diff=3672Overflow Filesystem2025-04-14T08:06:16Z<p>Weber: </p>
<hr />
<div>'''This page is obsolete as of Ubuntu 16.04'''<br />
<br />
<br />
==== overflow file system ====<br />
If the root file system is 100% full for some time, but the kernel or other system related jobs urgently need some space on the root file system, e.g., to create temp files in /tmp, the kernel creates a small '''overflow''' file system which is then mounted on /tmp:<br />
<pre><br />
Filesystem Size Used Avail Use% Mounted on<br />
overflow 1.0M 20K 1004K 2% /tmp<br />
</pre><br />
which resides in memory. Usually these overflow file systems are really tiny (1MB) - just large enough for urgent needs of the system.<br />
<br />
First of all you need to make space on the actual root file system, see the article on the [[Unmet Dependencies#completely_full_root_partition|completely full root file system]].<br />
<br />
Once you have freed enough space on the root file system, this overflow file system is no longer needed. However, since it usually contains several files in use by the system you can '''not''' simply unmount the file system. Instead of trying to figure out which files are open and trying to close them, it is in general much more straightforward to simple reboot the machine, once you have made sure that no important jobs are running on the machine:<br />
<pre><br />
shutdown -r now<br />
</pre><br />
<br />
[[Category:Admin]]</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3670Meggie Cluster2025-04-09T13:10:12Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
** Use ZFS zvols as storage backend<br />
** Find out which backend type is the best (file, block, SCSI passthrough)<br />
** Create a separate dataset for easier zfs setting propagation<br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
** Ca. 250W per node, 1kW per chassis<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3669Meggie Cluster2025-04-07T16:50:47Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
** Use ZFS zvols as storage backend<br />
** Create a separate dataset for easier zfs setting propagation<br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
** Ca. 250W per node, 1kW per chassis<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3668Meggie Cluster2025-04-07T13:42:03Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Test what happens on iSCSI connection problems<br />
** Switch malfunction<br />
** iSCSI server reboot<br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
** Use ZFS zvols as storage backend<br />
** Create a separate dataset for easier zfs setting propagation<br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3667New Network2025-04-06T15:47:48Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== MDI ===<br />
Ethernet on RJ45 ports consists of a pair of transmitting and receiving wires on both ends. Obviously the receiving pair on one end must be attached to the transmitting pair on the other end. In the old days it was important to consider this property when connecting devices via RJ45 twisted pairs, and in the appropriate case a [https://en.wikipedia.org/wiki/Ethernet_crossover_cable crossover cable] needed to be used. Nowadays ethernet iterfaces can automatically make sure to select the correct twisted pairs to transmit and receive data. This mechanism is called [https://en.wikipedia.org/wiki/Medium-dependent_interface#Automatic_MDI-X automatic MDI-X (automdix)]. It should be enabled on all ports of all switches, and probably is so by default.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB, webgui (defunct) ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB, webgui ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== HP switch administration ===<br />
==== Accessing via CLI ====<br />
Accessing the switches via browser is problematic for older switches since they use stuff like Java in the browser or flash. The CLI always works, if basic connectivity is ensured.<br />
<br />
===== telnet =====<br />
If the switch is up, ethernet is connected, IP address is assigned, subnets are configured correctly, and a computer is in the same VLAN (which should be the case), <code>telnet</code> can be used to access the switches. Just do <code>telnet {IP of switch}</code> and the you should reach the login of the CLI.<br />
<br />
===== serial =====<br />
Alternatively, and especially if the ethernet connection itself doesn't work (yet), the switches can be accessed directly by plugging a computer into the management port(s). Usually those are some kind of serial connection. Older models have a [https://en.wikipedia.org/wiki/D-subminiature D-Sub 9] connector for this purpose, newer models use a RJ45 port. In the latter case RJ45 is only the physical connector. It does not carry ethernet, only a [https://en.wikipedia.org/wiki/RS-232 RS-232] serial connection. Unless the computer itself has a serial port (unlikely these days), in both cases a serial to USB converter is required. Newer switches offer a USB port which internally has a USB to serial converter integrated and shows up in the connecting computer as a serial port. Regardless of the route choosen, if a serial to USB converter is attached the kernel log of the connecting computer should look something like this:<br />
<pre><br />
[ +0,042064] usbcore: registered new interface driver usbserial_generic<br />
[ +0,000009] usbserial: USB Serial support registered for generic<br />
[ +0,001588] usbcore: registered new interface driver ftdi_sio<br />
[ +0,000008] usbserial: USB Serial support registered for FTDI USB Serial Device<br />
[ +0,000101] ftdi_sio 1-4:1.0: FTDI USB Serial Device converter detected<br />
[ +0,000023] usb 1-4: Detected FT232RL<br />
[ +0,006308] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0<br />
</pre><br />
This means the serial port is available under <code>/dev/ttyUSB0</code>. Of course ne number in the end can change, depending on how many serial ports are present. Some devices show up as <code>/dev/ttyACM0</code> instead. These ports can be used with <code>screen</code> to talk to the switch, for example <code>screen /dev/ttyUSB0</code>. If everything goes well a message like <code>Connected with baud rate 9600</code> should show up.<br />
<br />
===== Authentication =====<br />
Depending on the model a username must be entered to access the switch. The "root" user for HP switches is usually called "manager". A password is required to log in in a mode which allows configuration changes. It should be set to our current LDAP admin password. If the password is not or incorrectly entered on the serial console the switches usually drops back to an unpriviledged user which can only look around.<br />
<br />
===== Entering configuration =====<br />
Before anything can be changed in the switch the configuration submenu must be entered. This can be done using the <code>config</code> command:<br />
<pre><br />
switch1# config<br />
switch1(config)# <br />
</pre><br />
Tab completion should work in all circumstances and is often very informative. An ad-hoc set of suggestions and help messages can also be triggered by putting a "?" in the terminal. Note that no "enter" is required in this case. As soon as the "?" is sent the terminal shows the suggestions.<br />
<br />
===== Saving the configuration =====<br />
Note that when changing the configuration of a switch only the running configuration is modified. Changes are not persistent over a reboot of the device. To make the changes persistent the configuration needs to be written to the flash memory of the device. This can be done by <code>write memory</code>.<br />
<br />
Interestingly, <code>write terminal</code> shows all commands necessary to achieve the currently running configuration of the switch.<br />
<br />
===== Basic setup =====<br />
The basic switch setup can be accessed even from outside the <code>config</code> menu using the <code>setup</code> command. It's an interactive menu which is rather intuitive. Among other things, it can be used to set<br />
* The hostname of the switch<br />
* The password for the <code>manager</code> user<br />
* The default gateway<br />
* The IP of the switch in the [[#VLAN|default VLAN]]<br />
<br />
===== Port information =====<br />
Information about the interfaces of the switch can be retrieven by the <code>show interfaces</code> command which lists all interfaces and their information in a table. It can be followed by another option to determine the type of information in the table. These are the options:<br />
* (no option): Sent/received traffic, errors, drops, [https://en.wikipedia.org/wiki/Ethernet_flow_control flow control], broadcast limit<br />
* <code>brief</code> Port type, intrusion alert, enabled/disabled, port status (up/down), mode (speed, duplex), flow control, broadcast limit<br />
* <code>config</code> Port type, enabled/disabled, mode, flow control, [https://en.wikipedia.org/wiki/Medium-dependent_interface MDI] mode<br />
* <code>port-utilization</code> mode (speed, duplex), Rx data, Tx data<br />
Much more detailed ethernet information about specific ports can be obtained by <code>show interfaces ethernet {PORTS}</code>. The ports can be referred to as single ports, a separated list by commas, or ranges separated by "-", or combinations of all. <code>all</code> can be used to refer to all ports at once.<br />
<br />
===== VLAN =====<br />
Information about the configure VLANs can be obtained by the <code>show vlans</code> command. In our case this should only list the default VLAN. VLANs can be configure by entering the context of a specific VLAN, for example <code>vlan 10</code>. In this submenu ports can be assigned to this VLAN as untagged access ports by issuing <code>untagged {PORTS}</code>.<br />
<br />
===== Spanning tree =====<br />
STP can be enabled from the config menu via <code>spanning-tree enable</code>. RSTP can be enforced as the protocol to use via <code>spanning-tree force-version rstp-operation</code>. Port based information about the established tree can be retrieved via <code>show spanning-tree</code>. This gives information about the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Path_cost path cost], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Configuration port priority], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states port state], and designated bridge.<br />
<br />
===== Time =====<br />
The timezones in the HP world are odd. They are given in offsets in minutes. So the timezone of the observatory is +60. It can be set via <code>time timezone 60</code>. The switches can also accept different daylight saving rules. The correct one can be set via <code>time daylight-time-rule Middle-Europe-and-Portugal</code>. The switches are configured to retrieve their time via SNTP, by setting <code>timesync sntp</code>. It utilizes the normal [https://en.wikipedia.org/wiki/Network_Time_Protocol NTP] but has a much simpler implementation on the client side. The switches are configured to use the RRZE NTP servers directly via their IP. This is discouraged, but since the switches don't do DNS not otherwise possible. Information about the current SNTP settings can be retrieved via <code>show sntp</code>. SNTP servers can be set via <code>sntp server {IP}</code> on older models, and <code>sntp server priority 1 {IP}</code> on newer models. The current time can be displayed via <code>time</code>.<br />
<br />
===== MDI Mode =====<br />
The MDI mode can be set via <code>interface {PORT} mdix-mode {MODE}</code> where PORT is a list of ports and MODE is <code>mdi</code>, <code>mdix</code>, or <code>automdix</code>. Obviously we want the latter on all interfaces. To set all ports to auto MDI-X: <code>interface all mdix-mode automdix</code><br />
<br />
==== Accessing via a browser ====<br />
Some of the switches come with an integrated webserver which can be used to access information and perform configuration. While this seems convenient at first, in reality these servers rely on stuff like [https://en.wikipedia.org/wiki/Java_(programming_language) Java] or sometimes even [https://en.wikipedia.org/wiki/Adobe_Flash Adobe Flash] in the browser which is discontinued and simply does not work at all. Therefore using the CLI is the more reliable way.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3666New Network2025-04-06T14:35:20Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB, webgui (defunct) ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB, webgui ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== HP switch administration ===<br />
==== Accessing via CLI ====<br />
Accessing the switches via browser is problematic for older switches since they use stuff like Java in the browser or flash. The CLI always works, if basic connectivity is ensured.<br />
<br />
===== telnet =====<br />
If the switch is up, ethernet is connected, IP address is assigned, subnets are configured correctly, and a computer is in the same VLAN (which should be the case), <code>telnet</code> can be used to access the switches. Just do <code>telnet {IP of switch}</code> and the you should reach the login of the CLI.<br />
<br />
===== serial =====<br />
Alternatively, and especially if the ethernet connection itself doesn't work (yet), the switches can be accessed directly by plugging a computer into the management port(s). Usually those are some kind of serial connection. Older models have a [https://en.wikipedia.org/wiki/D-subminiature D-Sub 9] connector for this purpose, newer models use a RJ45 port. In the latter case RJ45 is only the physical connector. It does not carry ethernet, only a [https://en.wikipedia.org/wiki/RS-232 RS-232] serial connection. Unless the computer itself has a serial port (unlikely these days), in both cases a serial to USB converter is required. Newer switches offer a USB port which internally has a USB to serial converter integrated and shows up in the connecting computer as a serial port. Regardless of the route choosen, if a serial to USB converter is attached the kernel log of the connecting computer should look something like this:<br />
<pre><br />
[ +0,042064] usbcore: registered new interface driver usbserial_generic<br />
[ +0,000009] usbserial: USB Serial support registered for generic<br />
[ +0,001588] usbcore: registered new interface driver ftdi_sio<br />
[ +0,000008] usbserial: USB Serial support registered for FTDI USB Serial Device<br />
[ +0,000101] ftdi_sio 1-4:1.0: FTDI USB Serial Device converter detected<br />
[ +0,000023] usb 1-4: Detected FT232RL<br />
[ +0,006308] usb 1-4: FTDI USB Serial Device converter now attached to ttyUSB0<br />
</pre><br />
This means the serial port is available under <code>/dev/ttyUSB0</code>. Of course ne number in the end can change, depending on how many serial ports are present. Some devices show up as <code>/dev/ttyACM0</code> instead. These ports can be used with <code>screen</code> to talk to the switch, for example <code>screen /dev/ttyUSB0</code>. If everything goes well a message like <code>Connected with baud rate 9600</code> should show up.<br />
<br />
===== Authentication =====<br />
Depending on the model a username must be entered to access the switch. The "root" user for HP switches is usually called "manager". A password is required to log in in a mode which allows configuration changes. It should be set to our current LDAP admin password. If the password is not or incorrectly entered on the serial console the switches usually drops back to an unpriviledged user which can only look around.<br />
<br />
===== Entering configuration =====<br />
Before anything can be changed in the switch the configuration submenu must be entered. This can be done using the <code>config</code> command:<br />
<pre><br />
switch1# config<br />
switch1(config)# <br />
</pre><br />
Tab completion should work in all circumstances and is often very informative. An ad-hoc set of suggestions and help messages can also be triggered by putting a "?" in the terminal. Note that no "enter" is required in this case. As soon as the "?" is sent the terminal shows the suggestions.<br />
<br />
===== Saving the configuration =====<br />
Note that when changing the configuration of a switch only the running configuration is modified. Changes are not persistent over a reboot of the device. To make the changes persistent the configuration needs to be written to the flash memory of the device. This can be done by <code>write memory</code>.<br />
<br />
Interestingly, <code>write terminal</code> shows all commands necessary to achieve the currently running configuration of the switch.<br />
<br />
===== Basic setup =====<br />
The basic switch setup can be accessed even from outside the <code>config</code> menu using the <code>setup</code> command. It's an interactive menu which is rather intuitive. Among other things, it can be used to set<br />
* The hostname of the switch<br />
* The password for the <code>manager</code> user<br />
* The default gateway<br />
* The IP of the switch in the [[#VLAN|default VLAN]]<br />
<br />
===== Port information =====<br />
Information about the interfaces of the switch can be retrieven by the <code>show interfaces</code> command which lists all interfaces and their information in a table. It can be followed by another option to determine the type of information in the table. These are the options:<br />
* (no option): Sent/received traffic, errors, drops, [https://en.wikipedia.org/wiki/Ethernet_flow_control flow control], broadcast limit<br />
* <code>brief</code> Port type, intrusion alert, enabled/disabled, port status (up/down), mode (speed, duplex), flow control, broadcast limit<br />
* <code>config</code> Port type, enabled/disabled, mode, flow control, [https://en.wikipedia.org/wiki/Medium-dependent_interface MDI] mode<br />
* <code>port-utilization</code> mode (speed, duplex), Rx data, Tx data<br />
Much more detailed ethernet information about specific ports can be obtained by <code>show interfaces ethernet {PORTS}</code>. The ports can be referred to as single ports, a separated list by commas, or ranges separated by "-", or combinations of all.<br />
<br />
===== VLAN =====<br />
Information about the configure VLANs can be obtained by the <code>show vlans</code> command. In our case this should only list the default VLAN. VLANs can be configure by entering the context of a specific VLAN, for example <code>vlan 10</code>. In this submenu ports can be assigned to this VLAN as untagged access ports by issuing <code>untagged {PORTS}</code>.<br />
<br />
===== Spanning tree =====<br />
STP can be enabled from the config menu via <code>spanning-tree enable</code>. RSTP can be enforced as the protocol to use via <code>spanning-tree force-version rstp-operation</code>. Port based information about the established tree can be retrieved via <code>show spanning-tree</code>. This gives information about the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Path_cost path cost], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Configuration port priority], [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states port state], and designated bridge.<br />
<br />
===== Time =====<br />
The timezones in the HP world are odd. They are given in offsets in minutes. So the timezone of the observatory is +60. It can be set via <code>time timezone 60</code>. The switches can also accept different daylight saving rules. The correct one can be set via <code>time daylight-time-rule Middle-Europe-and-Portugal</code>. The switches are configured to retrieve their time via SNTP, by setting <code>timesync sntp</code>. It utilizes the normal [https://en.wikipedia.org/wiki/Network_Time_Protocol NTP] but has a much simpler implementation on the client side. The switches are configured to use the RRZE NTP servers directly via their IP. This is discouraged, but since the switches don't do DNS not otherwise possible. Information about the current SNTP settings can be retrieved via <code>show sntp</code>. SNTP servers can be set via <code>sntp server {IP}</code> on older models, and <code>sntp server priority 1 {IP}</code> on newer models. The current time can be displayed via <code>time</code>.<br />
<br />
==== Accessing via a browser ====<br />
Some of the switches come with an integrated webserver which can be used to access information and perform configuration. While this seems convenient at first, in reality these servers rely on stuff like [https://en.wikipedia.org/wiki/Java_(programming_language) Java] or sometimes even [https://en.wikipedia.org/wiki/Adobe_Flash Adobe Flash] in the browser which is discontinued and simply does not work at all. Therefore using the CLI is the more reliable way.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3665New Network2025-04-06T13:17:26Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
=== NTP ===<br />
All switches are configured to use the RRZE NTP servers directly via their IP addresses because they can't do DNS. <br />
<br />
=== Current switches ===<br />
Here is a list of the currently installed switches, their configuration, uplink, and some notes. All passwords are the LDAP password.<br />
{| class="wikitable"<br />
|+ Switches<br />
|-<br />
! name !! IP !! Usecase !! Model !! Connectivity !! Administration !! Notes<br />
|-<br />
| switch1 || 10.10.1.1 || 1G RJ45 distribution racks || HP ProCurve J4904A 2848 || 44x1G RJ45, 4x1G RJ45/SFP || telnet, D-Sub 9 + RS232-USB ||<br />
* Kind of old<br />
* No username required for login<br />
* Ports 45-48 are dual personality ports, either RJ45 or SFP can be used, not both.<br />
|-<br />
| switch2 || 10.10.1.2 || server management (iLO, iDRAC, ...) || HP J9781A 2530-48 || 48x100M RJ45, 2x1G RJ45/SFP || telnet, micro-USB B, RJ45 + RS232-USB ||<br />
* Username: manager<br />
* Ports 49-50 (or 49-52) are dual personality ports, either RJ45 or SFP can be used, not both<br />
|}<br />
<br />
=== HP switch configuration ===<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3664New Network2025-04-06T11:49:56Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the ''internal'' IP address, but the port is changed to 8443: https://10.10.0.1:8443. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access. SSH is is disabled, but the port is changed to 12345.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3663New Network2025-04-06T11:36:26Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
We could try just using NAT and port forward the necessary traffic. The webserver would be in our private network only, the router has the public IP and therefore FQDN of the webserver, and ports 80 and 443 are forwarded from the router to the webserver. This way no fiddling with virtualbox, mutliple IPs, etc. is necessary.<br />
<br />
When doing it this way access via SSH is also very easy. Forward port 22 to a server behind the router which acts as the login node. Then all SSH logins are done directly through <code>user@www.stern...</code>. No two ethernet connections required. <br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3662New Network2025-04-06T11:26:06Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* Check whether/how we can use only one domain, instead of two (internal.sternwarte ...)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
Maybe it's possible to somehow only use one domain spreading over two subnets, but since some are then public with public DNS records and some are not this might be difficult.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3661New Network2025-04-06T11:21:03Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switches and set a high spanning tree priority on the root bridge<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3660New Network2025-04-05T22:31:23Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Enable RSTP on all switchesa and set a high spanning tree priority on the root bridge<br />
* <s>Have only one DHCP server</s><br />
** <s><code>dnsmasq</code> does DHCP, DNS, and TFTP. We have two DHCP servers right now which is a problem. See the dhcp-authoritative)</s><br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
All switches should support RSTP (IEEE 802.1w).<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3659New Network2025-04-05T22:29:51Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Set a high spanning tree priority on the root bridge<br />
* <s>Have only one DHCP server</s><br />
** <s><code>dnsmasq</code> does DHCP, DNS, and TFTP. We have two DHCP servers right now which is a problem. See the dhcp-authoritative)</s><br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
=== Spanning tree ===<br />
We know for a fact that sometimes people plug ethernet cables into random ports, even both ends of it, causing potential loops (see mail on 22 August 2023). To avoid such loops the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol spanning tree protocol (STP)] defined in [https://en.wikipedia.org/wiki/IEEE_802.1D IEEE 802.1d]. When this protocol is enabled on a switch it exchanges [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Bridge_protocol_data_units bridge protocol data units (BPDUs)] with its neighbouring switches, containing some information about the network topology. This information is used to elect the so called [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Root_bridge_and_the_bridge_ID root bridge]. This switch can be thought of as the top of the spanning tree. All other switches then determine one port they use as a path towards this root bridge, even if it goes through another switch. All other ports which also provide a path towards the root bridge are disabled (set to the blocking [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Port_states state]). The election of the root bridge also enters an individual priority for each switch which can be set by the administrator. We should make sure to set a high priority on our central switch.<br />
<br />
When a change in the network topology is detected via the BPDUs 802.1d causes the entire spanning tree to be newly established with a new election of a root bridge. This can take somewhere around a minute, which is too long. Therefore the [https://en.wikipedia.org/wiki/Spanning_Tree_Protocol#Rapid_Spanning_Tree_Protocol rapid STP (RSTP)] was defined in IEEE 802.1w. It is fully backwards compatible with 802.1d and uses additional information to define port based rules such as alternate and backup ports for the path to the root bridge such that a topology change can be acted on in just a few seconds. It has the nice benefit that a link to a computer which is plugged into an access port on a given switch becomes ready in just a few seconds. We therefore enable RSTP on all our switches. Usually this is done by enabling STP and forcing the STP mode to RSTP. No further configuration is needed, except the priority of the root bridge.<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3658New Network2025-04-05T21:57:54Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Have only one DHCP server<br />
** <code>dnsmasq</code> does DHCP, DNS, and TFTP. We have two DHCP servers right now which is a problem. See the dhcp-authoritative)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. We can allocate a <code>/16</code> network from this block. Since it sounds nice we can choose the second octet to also be 10. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
=== VLAN ===<br />
[https://en.wikipedia.org/wiki/VLAN Local Area Networks (VLAN)] are logical layer 2 networks which can spread across multiple switches. They can be used to separate different broadcast domains (ethernet networks) and usually only contain a single IP subnet. Under certain scenarios VLANs can offer performance, flexibility and security advantages. We don't want to segment our network, therefore performance and flexibility enhancements are not possible. Security is also of no concern. We therefore do not use VLANs.<br />
<br />
However, from a technical perspective this is impossible with layer 3 switches. By default all interfaces of a switch are assigned to the so called default VLAN. For most switch manufacturers this is VLAN 1, but it doesn't have to be. Most manufacturers also use VLAN 1 as the native VLAN. This means that all frames in VLAN 1 are sent untagged (without VLAN information) also to other switches, even over tagged ports (trunk ports). This ensures interoperability without changing settings.<br />
<br />
We are good as long as all switches we buy have default VLAN 1 (which can not be changed) and native VLAN 1 (which can be changed). In case we get a switch which does not have the default VLAN 1 we need to set the native VLAN of this switch also to the same VLAN, making all frames in this VLAN untagged, enabling communication with other switches.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3657New Network2025-04-05T20:35:27Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Have only one DHCP server<br />
** <code>dnsmasq</code> does DHCP, DNS, and TFTP. We have two DHCP servers right now which is a problem. See the dhcp-authoritative)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. If we put our own computers in a separate [https://de.wikipedia.org/wiki/Virtual_Local_Area_Network VLAN] to make mangement easier we can allocate a <code>/16</code> network from this block and use the second octet to denote the VLAN. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Layer 2 ==<br />
With the router working we can use our own ethernet switches behind it without interfering with the RRZE switches.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3656New Network2025-04-05T18:20:12Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* Have only one DHCP server<br />
** <code>dnsmasq</code> does DHCP, DNS, and TFTP. We have two DHCP servers right now which is a problem. See the dhcp-authoritative)<br />
* When all hosts are in the private network<br />
** Adjust router settings<br />
*** Block private and bogon networks on WAN interface<br />
*** Remove routes between public and private network<br />
*** Disable gui login from public network<br />
*** Disable DHC relay<br />
** Adjust DHCP settings<br />
*** Make the private network the default one (remove internal tags from config)<br />
*** Remove classless static route from public to private network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. If we put our own computers in a separate [https://de.wikipedia.org/wiki/Virtual_Local_Area_Network VLAN] to make mangement easier we can allocate a <code>/16</code> network from this block and use the second octet to denote the VLAN. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
<br />
== DNS ==<br />
=== Domains ===<br />
Our public domain is <code>sternwarte.uni-erlangen.de</code>. For the internal domain we don't need to register a domain, it doesn't make sense. Currently it is configure to <code>internal.sternwarte.uni-erlangen.de</code> which is clunky but accurate. It is configured in <code>dnsmasq</code> which has the nice sideeffect of <code>bla.internal</code> being correctly resolved from computers in the public network to the computer in the private network. The other way round it does not work, but <code>sternwarte.uni-erlangen.de</code> is therefore configured in <code>dnsmasq</code> as a search domain for all hosts in the private network. If we make sure to not duplicate computer names in the private and public network DNS enables the reference of another computer directly by its hostname.<br />
<br />
=== /etc/hosts ===<br />
With all the hostnames available via the <code>dnsmasq</code> configuration there is no need to out them all in <code>/etc/hosts</code> as well. <code>dnsmasq</code> knows the IP associated with a hostname from its dhcp configuration file and uses it to resolve the names if a DNS query is made. We could therefore only put aliases (such as <code>www</code>) in the <code>/etc/hosts</code> file and save the work of always trying to keep <code>/etc/hosts</code> up to date.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3655New Network2025-04-05T17:33:20Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* Check whether the router is set to switch on after power failure in the BIOS<br />
* When all hosts are in the private network: Adjust router settings<br />
** Block private and bogon networks on WAN interface<br />
** Remove routes between public and private network<br />
** Disable gui login from public network<br />
** Disable DHC relay<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At least one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. If we put our own computers in a separate [https://de.wikipedia.org/wiki/Virtual_Local_Area_Network VLAN] to make mangement easier we can allocate a <code>/16</code> network from this block and use the second octet to denote the VLAN. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE (WAN) and has a public IP, the other one is connected to our own switches and has a private ip (LAN). Both interfaces are labelled in the back of the server. The router routes traffic between these two networks to make our computers talk to each other.<br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
==== Gateway ====<br />
The router uses the RRZE gateway as its own upstream gateway. By default the firmware always sends packets destined to the WAN network to this gateway. At the time of writing this is problematic because the upstream gateway drops packets arriving from a private network (according to RFC1918) which makes it impossible for hosts in our private network to directly communicate with hosts in our public network. This option is therefore disabled in the configuration of the router. See the "Disable force gateway" option.<br />
<br />
==== Private and bogon networks ====<br />
According to RFC1918 packets with source or destination IP addresses in private networks must not be routed through the internet. It often happens that such packets are generated and end up at edge routers (keep the private notebooks connected to our network in mind). Therefore such routers usually drop these packets on purpose. At the time of writing we actually want to route such packets from our private to our public network and vice versa. Therefore the option to drop packets from or to private networks is disabled on both interfaces. When we have all our hosts behind the router we should re enable this option for the WAN interface to avoid trouble with the RRZE. On the LAN interface this option should obviously stay disabled.<br />
<br />
There are addresses in the public IPv4 space which are not assigned to anything or do not make sense. These are called [https://de.wikipedia.org/wiki/Bogon "Bogon" networks]. The router is configured to drop these packets if they arrive on the WAN interface.<br />
<br />
==== NAT ====<br />
Hosts within the private network can not directly talk to the internet (see the previous section), they need an intermidiary to do this. This is accomplished by using [https://en.wikipedia.org/wiki/Network_address_translation Network address translation (NAT)]. When a client wants to talk to the internet (except our own public network, see the following section), the router replaces the source address of the packet with its own public address and sends it of to the WAN. From the WAN it looks as if the router is making requests, instead of the client. When the reply arrives on the WAN interface the router changes the destination address of the packet from its own public address to the private address of the client and sends if off into the LAN.<br />
<br />
We need this mechanism for all clients behind the router to talk to the internet, except our own public network. This option is available as outbound NAT in the routers firmware and enabled by default.<br />
<br />
There is an important exception: Traffic from our private network which has a destination in our own public network should be routed directly, without any NAT, such that these two computers can talk directly to each other. Therefore there is a rule in the outbound NAT section which identifies such traffic and disables NAT for it. It can then be routed normally (see the following section). Note that this should be disabled as soon as all our computers are in the private network and can talk to each other directly.<br />
<br />
==== Routing between public and private network ====<br />
At the time of writing our computers are distributed in a private and a public network. Both need to be reachable by each other. Therefore the router allows traffic to pass between these exact two networks without any restrictions. The rules for this can be found in the LAN and WAN sections of the firewall rules of the router. Especially the rule allowing any traffic originating from the public network to pass directly into the private network should be disabled as soon as all computers are in the private network.<br />
<br />
==== DHCP relay ====<br />
At the time of writing we have DHCP (and DNS etc.) servers in the public network. Replicating these services behind the router can be avoided by allowing the router to forward DHCP requests from the private to the public network. This is done in the DHC relay section in the Services menu. With this service and the working routing and NAT there is no need to replicate these services in the private network for now. Everything seems to work, including PXE boot to install clients within the private network. As soon as all computers are in the private network the DHC relay should be disabled.<br />
<br />
==== Backup ====<br />
The configuration can be backed up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
=== Updates ===<br />
Updates of the route can simply be done through the GUI. For security reasons this is recommended on a regular basis, but since we have the firewall of the RRZE in front of it it's probably not critical. Often the updates ca be done even without restarting the firmware.<br />
<br />
== Remote Access ==<br />
We need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
== Exposed services ==<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
== Miscellaneous Settings/Optimizations ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3654New Network2025-04-05T16:41:42Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
We're creating our own network inside a DMZ.<br />
<br />
== To Do List ==<br />
* When all hosts are in the private network: Adjust router settings<br />
** Block private and bogon networks on WAN interface<br />
** Remove routes between public and private network<br />
** Disable gui login from public network<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At lesat one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. If we put our own computers in a separate [https://de.wikipedia.org/wiki/Virtual_Local_Area_Network VLAN] to make mangement easier we can allocate a <code>/16</code> network from this block and use the second octet to denote the VLAN. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution which is very easy to set up and highly reliable. The server has two 1G interfaces. One is connected to the public network managed by the RRZE and has a public IP, the other one is connected to our own switches and has a private ip. The router routes traffic between these two networks to make our computers talk to each other. On the public interface neither private nor bogon networks are blocked. <br />
<br />
=== Configuration ===<br />
The router can be configure through the web gui. It's accessible directly over the IP address. The user for the login is <code>root</code> and the password is the same as the LDAP admin passwort (as of writing). A lot of information is provided in the [https://docs.opnsense.org/ OPNSense documentation]. One of the interfaces has a public IP, at the time of writing the public IP is <code>131.188.151.92</code>. The other interface has the private used as the gateway, <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
The configuration can be backup up simply by downloading it in the GUI. It's recommended to do this once in a while just in case the router dies. This configuration can then be very easily applied after a new installation.<br />
<br />
<br />
<br />
<br />
=== Gateway ===<br />
We can use a small dedicated server with at least two interfaces running a router distribution as gateway and firewall.<br />
pfSense and OPNSense are the most commonly used distributions for this workload.<br />
OPNSense seems to be the preferred choice. PW has a bit of experience with it (personal use).<br />
<br />
=== Port Forwarding ===<br />
Port forwarding would enable us to directly expose services to the WAN. For example dynamically redirect the SSH access to the login node or the gitlab server. Makes moving services '''much''' easier. But for this to work the RRZE needs to completely remove the firewall on this host.<br />
<br />
=== Remote Access ===<br />
Ideally we could use the option described above. Otherwise we need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
=== Exposed services ===<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. In principle this could also be done with port forwarding, but it's very ugly because the gateway then needs to have more public IP addresses. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
<br />
== Miscellaneous Settings/Advantages ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3653New Network2025-04-05T16:08:11Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
We're creating our own network inside a DMZ.<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At lesat one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. If we put our own computers in a separate [https://de.wikipedia.org/wiki/Virtual_Local_Area_Network VLAN] to make mangement easier we can allocate a <code>/16</code> network from this block and use the second octet to denote the VLAN. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
<br />
<br />
== Router ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution. It can very easily be configure through a webinterface. The server has two 1G interfaces. One is connected to the public network managed by the RRZE and has a public IP, the other one is connected to our own switches and has a private ip. At the time of writing the public IP is <code>131.188.151.92</code>, and the private IP is <code>10.10.0.1</code>. Its public IP might be changed depending on how we proceed with mechansim for remote access.<br />
<br />
<br />
<br />
<br />
=== Gateway ===<br />
We can use a small dedicated server with at least two interfaces running a router distribution as gateway and firewall.<br />
pfSense and OPNSense are the most commonly used distributions for this workload.<br />
OPNSense seems to be the preferred choice. PW has a bit of experience with it (personal use).<br />
<br />
=== Port Forwarding ===<br />
Port forwarding would enable us to directly expose services to the WAN. For example dynamically redirect the SSH access to the login node or the gitlab server. Makes moving services '''much''' easier. But for this to work the RRZE needs to completely remove the firewall on this host.<br />
<br />
=== Remote Access ===<br />
Ideally we could use the option described above. Otherwise we need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
=== Exposed services ===<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. In principle this could also be done with port forwarding, but it's very ugly because the gateway then needs to have more public IP addresses. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
<br />
== Miscellaneous Settings/Advantages ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3652New Network2025-04-05T16:04:25Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
We're creating our own network inside a DMZ.<br />
<br />
== Subnets ==<br />
=== Public ===<br />
The public subnet is a part of the internet as organized through the [https://de.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority IANA]. "Our" subnet is <code>131.188.151.0/24</code>. At lesat one of the IPs in this subnet is used by the RRZE to provide a gateway. It's a [https://www.cisco.com/site/de/de/index.html Cisco] router in the basement with the IP <code>131.188.151.1</code> and FQDN <code>astro.gate.uni-erlangen.de</code>. We need to use this gateway to access the WAN.<br />
<br />
=== Private ===<br />
We need to use a private network as defined in [https://www.rfc-editor.org/rfc/rfc1918.html RFC1918] for our own subnet. Because it's easy to remember and short to write we pick the class A network from RFC1918 which is <code>10.0.0.0/8</code>. If we put our own computers in a separate [https://de.wikipedia.org/wiki/Virtual_Local_Area_Network VLAN] to make mangement easier we can allocate a <code>/16</code> network from this block and use the second octet to denote the VLAN. This gives us more than 65000 IP addresses to work with in <code>10.10.0.0/16</code>. Because it makes sense we use the first address for our own gateway: <code>10.10.0.1</code><br />
<br />
==== Address blocks ====<br />
Using the <code>/16</code> doesn't only provide more than enough addresses, probably forever, it also enables the grouping into blocks of addresses which denote different classes of devices. This makes it easier to identify the devices and find their IP address. Note that these are not separate subnets, it's only nomenclature. These blocks are used in the DHCP server of [https://thekelleys.org.uk/dnsmasq/doc.html dnsmasq] and defined in the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet/-/blob/master/modules/remeis/files/dnsmasq/internal-dhcp.conf internal-dhcp.conf] file of the [https://www.sternwarte.uni-erlangen.de/gitlab/Admins/puppet puppet] gitlab repository.<br />
<br />
{| class="wikitable"<br />
|+ Address blocks<br />
|-<br />
! Block !! Device class<br />
|-<br />
| 10.10.1.* || Switches<br />
|-<br />
| 10.10.2.* || Management interfaces (iLO, iDRAC, ...)<br />
|-<br />
| 10.10.10.* || Servers<br />
|-<br />
| 10.10.20.* || VMs<br />
|-<br />
| 10.10.30.* || Meggie cluster<br />
|-<br />
| 10.10.40.* || Blade center<br />
|-<br />
| 10.10.50.* || Messier cluster<br />
|-<br />
| 10.10.90.* || Functional hosts (Raspbbery Pis, ...)<br />
|-<br />
| 10.10.100.* || Office computers main building<br />
|-<br />
| 10.10.200.* || Testing (Installation testing, ...)<br />
|-<br />
| 10.10.250.* || Private notebooks<br />
|}<br />
<br />
<br />
<br />
== Uplink ==<br />
<br />
The uplink to the WAN is realized with a server (Dell Poweredge 1950) which runs [https://opnsense.org OPNSense], a FreeBSD based routing and firewall distribution. It can very easily be configure through a webinterface. The server has two 1G interfaces. One is connected to the public network managed by the RRZE and has a public IP, the other one is connected to our own switches and has a private ip. At the time of writing the public IP is <code>131.188.151.92</code>, and the private IP is <code>10.10.0.1</code>.<br />
<br />
=== Gateway ===<br />
We can use a small dedicated server with at least two interfaces running a router distribution as gateway and firewall.<br />
pfSense and OPNSense are the most commonly used distributions for this workload.<br />
OPNSense seems to be the preferred choice. PW has a bit of experience with it (personal use).<br />
<br />
=== Port Forwarding ===<br />
Port forwarding would enable us to directly expose services to the WAN. For example dynamically redirect the SSH access to the login node or the gitlab server. Makes moving services '''much''' easier. But for this to work the RRZE needs to completely remove the firewall on this host.<br />
<br />
=== Remote Access ===<br />
Ideally we could use the option described above. Otherwise we need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
=== Exposed services ===<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. In principle this could also be done with port forwarding, but it's very ugly because the gateway then needs to have more public IP addresses. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
<br />
== Miscellaneous Settings/Advantages ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=New_Network&diff=3651New Network2025-04-05T15:10:36Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
We're thinking about creating our own network inside a DMZ.<br />
<br />
== Uplink ==<br />
<br />
=== Gateway ===<br />
We can use a small dedicated server with at least two interfaces running a router distribution as gateway and firewall.<br />
pfSense and OPNSense are the most commonly used distributions for this workload.<br />
OPNSense seems to be the preferred choice. PW has a bit of experience with it (personal use).<br />
<br />
=== Port Forwarding ===<br />
Port forwarding would enable us to directly expose services to the WAN. For example dynamically redirect the SSH access to the login node or the gitlab server. Makes moving services '''much''' easier. But for this to work the RRZE needs to completely remove the firewall on this host.<br />
<br />
=== Remote Access ===<br />
Ideally we could use the option described above. Otherwise we need a login node with a separate interface and a dedicated public IP, see the following section.<br />
<br />
=== Exposed services ===<br />
Some services need to be exposed to the internet under a specific address. Most importantly the webserver (www.stern....de). This needs to have a public IP resolving to a specific host. In principle this could also be done with port forwarding, but it's very ugly because the gateway then needs to have more public IP addresses. Therefore such services need to run on a server which has two interfaces. One for the LAN and one for the WAN. The weak end system model of Linux ''might'' become a problem here.<br />
<br />
<br />
== Hardware ==<br />
Desired uplink speeds for individual hardware<br />
* NFS servers: At least 10G, some maybe with 2x10G.<br />
* Bladecenter: 10G<br />
* Meggie nodes: 80x1G<br />
* Other clients: 1G<br />
<br />
=== Main Building ===<br />
In the main building we can create our own layer 2 network fairly straightforwardly.<br />
<br />
Let's ''please'' mount the switches in reverse in the server racks.<br />
<br />
<br />
==== Root Bridge ====<br />
The root bridge will be the core of the network. Needs to have the capability to attach the NFS servers as well as other fast enough uplinks to switches, so at least several 10G SFP+ ports.<br />
<br />
Omada: SX3032F, 32x10G SFP+, ~900€<br />
<br />
Ubiquity (same as above): USW-Pro-Aggregation, 28x10G SFP+, 4x25G SFP28, ~900€<br />
<br />
<br />
==== Meggie Switches ====<br />
Each Meggie node provides a 1G uplink (apart from Intel OP). With 80 nodes we need 2 48 port switches.<br />
<br />
Omada: SG3452X, 48x1G RJ45, 4x10G SFP+, ~400€<br />
<br />
Ubiquity: USW-Pro-48, 48x1G RJ45, 4x10G SFP+, ~600€<br />
<br />
<br />
==== Peripheral Switches ====<br />
If we decommission the Messiers we can make due with 3 peripheral switches for the main building. If the RRZE lets us use the currently installed 3 switches with the 10G uplink we do not need new hardware for this.<br />
<br />
==== Shopping List ====<br />
* 10G root bridge: ~900 Euros<br />
* 2x 48x1G + 10G uplink meggie switches: 2x~400 Euros<br />
* Lots of Cat 6 (or 5e) cables for the meggies<br />
* We might need new DAC cables<br />
* Total: 1700 Euros<br />
<br />
<br />
<br />
== Integrating Meridian building and Bundschuhhaus ==<br />
<br />
=== Layer 2 ===<br />
There are two fibers going to the Meridian building. One is patched through to the Bundschuhhaus.<br />
For the Meridian building it is technically possible to use one of the fibers to directly integrate an additional switch in the Clock room via layer 2. But the RRZE needs to play ball for this because they need to switch the other pair through to Bundschuhhause. If this route is taken there is still no option to integrate Bundschuhhause on layer 2.<br />
<br />
=== Layer 3 ===<br />
A VPN would be an option to incorporate the other buildings over layer 3 without interaction with the RRZE (fingers crossed). Each building needs to have a separate OPNsense box establishing a VPN directly to the main gateway over the WAN and an attached switch for the computers.<br />
<br />
NFS etc. might be a bit wonky in this scenarion. Performance will certainly be limited. Strong CPU hardware of the OPNSense boxes is required for this to work with low latency.<br />
<br />
==== IPSec ====<br />
According to the documentation this is the preferred solution for a site-to-site setup such as in our case. It's a bit complicated to set up compared to the other options. It requires an open port on the main gateway (similar to port forwarding). The computers on the other sites then also need to be in a separate subnet, necessitating a separate DHCP (possibly DNS etc.) server (TBC).<br />
<br />
==== Wireguard ====<br />
Much easier to set up compared to IPSec, but might be less reliable. Unclear about the different subnet. Might not require an open port on the main gateway.<br />
<br />
==== OpenVPN ====<br />
Fairly simple to set up, running via SSL, can mimick HTTPS traffic. Requires an open port.<br />
<br />
<br />
== Miscellaneous Settings/Advantages ==<br />
* Enable jumbo frames for better NFS/iSCSI performance<br />
* Use spanning tree? If so configure the root bridge with high priority, and use portfast</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3650Meggie Cluster2025-04-05T10:53:16Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
** Use ZFS zvols as storage backend<br />
** Create a separate dataset for easier zfs setting propagation<br />
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted<br />
** Optimize dataset settings for iSCSI targets<br />
*** Compression lz4<br />
*** Larger <code>recordsize</code> (1M or something)<br />
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3649Meggie Cluster2025-04-04T23:42:34Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** <s>Make no scratch available</s><br />
** <s>Make /tmp and friends a tmpfs</s><br />
** <s>Restrict sizes of all tmpfs</s><br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3648Meggie Cluster2025-04-04T21:48:28Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
* Remove the stupid /swap.img file. This is a general point affecting all computers<br />
* Adjust puppet<br />
** Make no scratch available<br />
** Make /tmp and friends a tmpfs<br />
** Restrict sizes of all tmpfs<br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3647Meggie Cluster2025-04-04T21:47:18Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
* Adjust puppet<br />
** Make no scratch available<br />
** Make /tmp and friends a tmpfs<br />
** Restrict sizes of all tmpfs<br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3646Meggie Cluster2025-04-04T20:34:00Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
* Adjust puppet<br />
** Make no scratch available<br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=User:Weber&diff=3645User:Weber2025-04-04T09:33:44Z<p>Weber: Created page with "What an idiot"</p>
<hr />
<div>What an idiot</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3644Meggie Cluster2025-04-04T09:30:41Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* Make the installer recognize the iSCSI LUN as a block device before searching for those<br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
* Adjust puppet<br />
** Make no scratch available<br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3643Meggie Cluster2025-04-03T22:35:01Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
* Adjust puppet<br />
** Make no scratch available<br />
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weberhttps://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Meggie_Cluster&diff=3642Meggie Cluster2025-04-03T22:32:29Z<p>Weber: </p>
<hr />
<div>[[Category:Admin]]<br />
<br />
The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.<br />
<br />
== To Do List ==<br />
<br />
=== OS setup ===<br />
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s><br />
* <s>Supply LUN information directly to the UEFI via DHCP</s><br />
* Make a list of UEFI settings<br />
* Prepare LUNs<br />
<br />
=== Hardware setup ===<br />
* Install nodes in a rack<br />
* Buy and set up up switches<br />
* Lots of cabling<br />
* Power requirements?<br />
<br />
<br />
== The Boot Process ==<br />
<br />
=== General layout ===<br />
The boot process works like this:<br />
# UEFI stage<br />
## The UEFI starts up<br />
## Establishes a connection to the network<br />
## Runs a DHCP client<br />
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server<br />
## Accesses the GPT on the LUN and loads grub<br />
# grub stage<br />
## Grub starts<br />
## Loads the kernel and initrd from the LUN<br />
## Jumbs into the kernel<br />
# ramdisk stage<br />
## The kernel runs<br />
## Mounts the initrd<br />
## Inside the initrd our custom iSCSI hook is called<br />
### Loads the iSCSI kernel module<br />
### Loads the iSCSI iBFT kernel module<br />
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device<br />
# Normal boot continues<br />
<br />
<br />
<br />
=== Further explanations ===<br />
==== UEFI DHCP ====<br />
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.<br />
<br />
==== grub ====<br />
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.<br />
<br />
==== iBFT ====<br />
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.<br />
<br />
==== initrd modifications ====<br />
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:<br />
<source><br />
#!/bin/sh<br />
# iSCSI init script<br />
PREREQ=""<br />
prereqs()<br />
{<br />
echo "$PREREQ"<br />
}<br />
<br />
case $1 in<br />
prereqs)<br />
prereqs<br />
exit 0<br />
;;<br />
esac<br />
<br />
. /scripts/functions<br />
<br />
log_begin_msg "Begin iSCSI init"<br />
<br />
modprobe iscsi_tcp<br />
modprobe iscsi_ibft<br />
<br />
log_begin_msg "Network configuration based on iBFT"<br />
<br />
iscsistart -N || panic "Could not initialize iSCSI"<br />
<br />
log_begin_msg "Waiting to finish iscsistart"<br />
until iscsistart -b ; do<br />
sleep 1<br />
done<br />
</source><br />
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.<br />
<br />
=== Installing ===<br />
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.<br />
<br />
<br />
=== iSCSI targets ===<br />
We need to put the iSCSI Luns somewhere. We have 80 computers. The root filesystem size for each node will be around 40GB. In total around 3TB of data will be used for these LUNs, minus compression by ZFS. The new virgo is maybe a good choice for that, but it's still in testing mode. This needs to be clarified.<br />
<br />
== UEFI settings ==<br />
Settings applied after resetting the UEFI to default settings:<br />
* ...</div>Weber