Remeis-Wiki - User contributions [en]

Meggie Cluster

2025-09-10T07:45:57Z

Reinmann: /* OS setup */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* <s>Make a list of UEFI settings</s>
* <s>Prepare LUNs</s>
** <s>Use ZFS zvols as storage backend</s>
** <s>Find out which backend type is the best (file, block, SCSI passthrough)</s>
** <s>Create a separate dataset for easier zfs setting propagation</s>
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** <s>Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4</s>
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code>
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.
The root filesystem size for each node is 80.8GB.
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:

Enter the BIOS menu by pressing F2.

In the settings, first RESET everything to default.
Then those changes should be made:

<source>

Main
--> Quiet Boot: Disabled
Advanced
--> Processor Configuration
--> Intel Hyper Threading: Disabled
--> Power & Performance
--> CPU Power & Performance: Performance
--> CPU HWPM State Control
--> Enable CPU HWPM: HWPM Native Mode
--> PCI Configuration
--> NIC Configuration
--> NIC Port 2: Disabled
Server Management
--> Clear System Event Log: Clear it here
Advanced Boot Options
--> System Boot Timeout: 10
--> Bood Mode: UEFI
--> Boot Option Retry: Enabled

</source>

REBOOT the system (with e.g. ALT + CMD + DEL)
Then change again in the BIOS settings:

<source>
Advanced
--> PCI Configuration
--> UEFI Network Stack
-->IPv6 PXE Support: Disabled
--> UEFI Option ROM Control
--> IPv4 Network Configuration
--> Configured: [x]
--> Enable DHCP: [x]
--> iSCSI Configuration
--> iSCSI Initiator Name: Format is: iqn.1886.de.sternwarte.iscsi:meggie-x-y
where x: rack number (01 is top, 20 is the lowest rack);
y: |-----|-----| for one rack are 4 nodes.
| 1 | 2 |
|-----|-----|
| 3 | 4 |
|-----|-----|
eg: iqn.1886.de.sternwarte.iscsi:meggie-11-1 for node in rack 11 and place 1
--> Add an Attempt
--> MAC-address
--> iSCSI Mode: Enabled
--> Connection Retry Count: 2
--> Connection Establishing Timeout: 5000
--> Enable DHCP: [x]
--> Get target info via DHCP: [x]
--> Authentication Type: None
</source>

REBOOT and press F6 to enter the boot menu:

Go to IP based booting option (should be the last of the three)
Select partition:
<source>
Ubuntu 20.4 puppet iSCSI autopartition --DO NOT USE IF NO IDEA--
</source>

Be careful, because there exist also other versions like blank.

Meggie Cluster

2025-09-10T07:45:33Z

Reinmann: /* OS setup */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** <s>Switch malfunction</s>
** <s>iSCSI server reboot</s>
** <s>It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow</s>
* <s>Make a list of UEFI settings</s>
* <s>Prepare LUNs</s>
** <s>Use ZFS zvols as storage backend</s>
** <s>Find out which backend type is the best (file, block, SCSI passthrough)</s>
** <s>Create a separate dataset for easier zfs setting propagation</s>
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** <s>Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4</s>
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code>
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.
The root filesystem size for each node is 80.8GB.
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:

Enter the BIOS menu by pressing F2.

In the settings, first RESET everything to default.
Then those changes should be made:

<source>

Main
--> Quiet Boot: Disabled
Advanced
--> Processor Configuration
--> Intel Hyper Threading: Disabled
--> Power & Performance
--> CPU Power & Performance: Performance
--> CPU HWPM State Control
--> Enable CPU HWPM: HWPM Native Mode
--> PCI Configuration
--> NIC Configuration
--> NIC Port 2: Disabled
Server Management
--> Clear System Event Log: Clear it here
Advanced Boot Options
--> System Boot Timeout: 10
--> Bood Mode: UEFI
--> Boot Option Retry: Enabled

</source>

REBOOT the system (with e.g. ALT + CMD + DEL)
Then change again in the BIOS settings:

<source>
Advanced
--> PCI Configuration
--> UEFI Network Stack
-->IPv6 PXE Support: Disabled
--> UEFI Option ROM Control
--> IPv4 Network Configuration
--> Configured: [x]
--> Enable DHCP: [x]
--> iSCSI Configuration
--> iSCSI Initiator Name: Format is: iqn.1886.de.sternwarte.iscsi:meggie-x-y
where x: rack number (01 is top, 20 is the lowest rack);
y: |-----|-----| for one rack are 4 nodes.
| 1 | 2 |
|-----|-----|
| 3 | 4 |
|-----|-----|
eg: iqn.1886.de.sternwarte.iscsi:meggie-11-1 for node in rack 11 and place 1
--> Add an Attempt
--> MAC-address
--> iSCSI Mode: Enabled
--> Connection Retry Count: 2
--> Connection Establishing Timeout: 5000
--> Enable DHCP: [x]
--> Get target info via DHCP: [x]
--> Authentication Type: None
</source>

REBOOT and press F6 to enter the boot menu:

Go to IP based booting option (should be the last of the three)
Select partition:
<source>
Ubuntu 20.4 puppet iSCSI autopartition --DO NOT USE IF NO IDEA--
</source>

Be careful, because there exist also other versions like blank.

Meggie Cluster

2025-08-12T12:18:12Z

Reinmann: /* UEFI settings */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code>
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.
The root filesystem size for each node is 80.8GB.
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:

Enter the BIOS menu by pressing F2.

In the settings, first RESET everything to default.
Then those changes should be made:

<source>

Main
--> Quiet Boot: Disabled
Advanced
--> Processor Configuration
--> Intel Hyper Threading: Disabled
--> Power & Performance
--> CPU Power & Performance: Performance
--> CPU HWPM State Control
--> Enable CPU HWPM: HWPM Native Mode
--> PCI Configuration
--> NIC Configuration
--> NIC Port 2: Disabled
Server Management
--> Clear System Event Log: Clear it here
Advanced Boot Options
--> System Boot Timeout: 10
--> Bood Mode: UEFI
--> Boot Option Retry: Enabled

</source>

REBOOT the system (with e.g. ALT + CMD + DEL)
Then change again in the BIOS settings:

<source>
Advanced
--> PCI Configuration
--> UEFI Network Stack
-->IPv6 PXE Support: Disabled
--> UEFI Option ROM Control
--> IPv4 Network Configuration
--> Configured: [x]
--> Enable DHCP: [x]
--> iSCSI Configuration
--> iSCSI Initiator Name: Format is: iqn.1886.de.sternwarte.iscsi:meggie-x-y
where x: rack number (01 is top, 20 is the lowest rack);
y: |-----|-----| for one rack are 4 nodes.
| 1 | 2 |
|-----|-----|
| 3 | 4 |
|-----|-----|
eg: iqn.1886.de.sternwarte.iscsi:meggie-11-1 for node in rack 11 and place 1
--> Add an Attempt
--> MAC-address
--> iSCSI Mode: Enabled
--> Connection Retry Count: 2
--> Connection Establishing Timeout: 5000
--> Enable DHCP: [x]
--> Get target info via DHCP: [x]
--> Authentication Type: None
</source>

REBOOT and press F6 to enter the boot menu:

Go to IP based booting option (should be the last of the three)
Select partition:
<source>
Ubuntu 20.4 puppet iSCSI autopartition --DO NOT USE IF NO IDEA--
</source>

Be careful, because there exist also other versions like blank.

Meggie Cluster

2025-08-12T12:05:23Z

Reinmann: /* UEFI settings */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code>
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.
The root filesystem size for each node is 80.8GB.
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:

Enter the BIOS menu by pressing F2.

In the settings, first RESET everything to default.
Then those changes should be made:

<source>

Main
--> Quiet Boot: Disabled
Advanced
--> Processor Configuration
--> Intel Hyper Threading: Disabled
--> Power & Performance
--> CPU Power & Performance: Performance
--> CPU HWPM State Control
--> Enable CPU HWPM: HWPM Native Mode
--> PCI Configuration
--> NIC Configuration
--> NIC Port 2: Disabled
Server Management
--> Clear System Event Log: Clear it here
Advanced Boot Options
--> System Boot Timeout: 10
--> Bood Mode: UEFI
--> Boot Option Retry: Enabled

</source>

REBOOT the system (with e.g. ALT + CMD + DEL)
Then change again in the BIOS settings:

<source>
Advanced
--> PCI Configuration
--> UEFI Network Stack
-->IPv6 PXE Support: Disabled
</source>

Meggie Cluster

2025-08-12T11:56:04Z

Reinmann: /* iSCSI targets */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolume <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code>
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.
The root filesystem size for each node is 80.8GB.
Once created, they shouldn't be touched unless nodes are failing and we want to delete unnecessary images.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:
* ...

Meggie Cluster

2025-08-12T11:54:42Z

Reinmann: /* iSCSI targets */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolum <code> pool/iscsi </code> (called: <code>generate_zfs_images.sh</code>). Then the iscsi targets are being created with: <code>create_iscsi_targets.sh</code>
With <code>destroy_all_zfs_images.sh</code> everything can be reverted.
The root filesystem size for each node is 80.8GB.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:
* ...

Meggie Cluster

2025-08-12T11:53:57Z

Reinmann: /* iSCSI targets */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolum <source> pool/iscsi </source> (called: generate_zfs_images.sh). Then the iscsi targets are being created with: create_iscsi_targets.sh
With destroy_all_zfs_images.sh everything can be reverted.
The root filesystem size for each node is 80.8GB.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:
* ...

Meggie Cluster

2025-08-12T11:53:27Z

Reinmann: /* iSCSI targets */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolum <source> pool/iscsi <\source> (called: generate_zfs_images.sh). Then the iscsi targets are being created with: create_iscsi_targets.sh
With destroy_all_zfs_images.sh everything can be reverted.
The root filesystem size for each node is 80.8GB.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:
* ...

Meggie Cluster

2025-08-12T11:52:38Z

Reinmann: /* iSCSI targets */

[[Category:Admin]]

The meggies are diskless nodes inherited from the RRZE used from computing only. We would like to boot them from the network. NFS and iSCSI are two options to accomplish this. Since iSCSI is directly supported by the UEFI of the nodes this is the option which is by far the easiest to implement.

== Hardware ==
The main boards are S2600KPR from Intel. Four of those are in a 2U chassis, called H2000G. The rails for the installation are called AXXELVRAIL.

== To Do List ==

=== OS setup ===
* <s>Make the installer recognize the iSCSI LUN as a block device before searching for those</s>
* <s>Supply LUN information directly to the UEFI via DHCP</s>
* Test what happens on iSCSI connection problems
** Switch malfunction
** iSCSI server reboot
** It doesn't take long until the kernel gives up on the iSCSI target. We need to increase this timeout somehow
* Make a list of UEFI settings
* Prepare LUNs
** Use ZFS zvols as storage backend
** Find out which backend type is the best (file, block, SCSI passthrough)
** Create a separate dataset for easier zfs setting propagation
** Leave a README file in the dataset directory for other people to know that it shouldn't be deleted
** Optimize dataset settings for iSCSI targets
*** Compression lz4
*** Larger <code>recordsize</code> (1M or something)
** Name the zvols <code>zvol-MM-m</code> where MM is the chassis number from 01 to 20 and m the node number from 1 to 4
* Remove the stupid /swap.img file. This is a general point affecting all computers
* Adjust puppet
** <s>Make no scratch available</s>
** <s>Make /tmp and friends a tmpfs</s>
** <s>Restrict sizes of all tmpfs</s>
** Remove unnecessary user stuff? (e.g. x2go, firefox, etc.)

=== Hardware setup ===
* Install nodes in a rack
* Buy and set up up switches
* Lots of cabling
* Power requirements?
** Ca. 250W per node, 1kW per chassis

== The Boot Process ==

=== General layout ===
The boot process works like this:
# UEFI stage
## The UEFI starts up
## Establishes a connection to the network
## Runs a DHCP client
## Gets an IP as well as the iSCSI LUN information directly from our DHCP server
## Accesses the GPT on the LUN and loads grub
# grub stage
## Grub starts
## Loads the kernel and initrd from the LUN
## Jumbs into the kernel
# ramdisk stage
## The kernel runs
## Mounts the initrd
## Inside the initrd our custom iSCSI hook is called
### Loads the iSCSI kernel module
### Loads the iSCSI iBFT kernel module
### Runs <code>iscsistart</code> which makes the iSCSI LUN available as a block device
# Normal boot continues

=== Further explanations ===
==== UEFI DHCP ====
The UEFI can talk to the DHCP server and get an IP. We use DHCP option 17 (root-path) to supply the iSCSI information to the nodes.

==== grub ====
As of now we don't know why grub even works. It's installed on the efi partition on the LUN, just like a normal computer. The UEFI uses the information on this EFI partition to find grub and run it, and grub then sees the partitions of the installed OS. Either the UEFI somehow emulates a blockdevice for grub such that it can see these partitions and load the kernel and initrd, or grub somehow also does iSCSI by using the iBFT. Anyway, it loads the kernel and initrd, which is the most important part.

==== iBFT ====
This is really cool. When the UEFI launches and gets iSCSI information from the DHCP server it puts this information into the EFI system table, the Boot Firmware Table (iBFT). Linux can use this information to connect to the same LUN used for the boot process and set up the block device before trying to mount the root partition. The necessary kernel module is called <code>iscsi_ibft</code>. After the module is loaded its only a matter of calling the <code>iscsistart</code> program to set up the block device. All this needs to happen before root is mounted, therefore modifications to the initrd are necessary.

==== initrd modifications ====
During the boot, when the kernel is running and mounted the initrd, the <code>iscsi_ibft</code> and <code>iscsi_tcp</code> modules need to be loaded, after which the block device can be created. To do this a hook needs to be installed in the initramfs. On ubuntu this can be done by putting files into the <code>/etc/initramfs-tools/scripts</code> directory (or rather subdirectories). The hook for the iSCSI setup need to happen after the network is available, which means the hook needs to be put in the <code>local-top</code> directory and made executable. The content is:
<source>
#!/bin/sh
# iSCSI init script
PREREQ=""
prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /scripts/functions

log_begin_msg "Begin iSCSI init"

modprobe iscsi_tcp
modprobe iscsi_ibft

log_begin_msg "Network configuration based on iBFT"

iscsistart -N || panic "Could not initialize iSCSI"

log_begin_msg "Waiting to finish iscsistart"
until iscsistart -b ; do
sleep 1
done
</source>
Of course the script needs to be made executable. After that the initial ramdisk(s) need to be regenerated by calling <code>update-initramfs -u</code>. When the script is called during the boot process the block device exists for the kernel to mount the root filesystem and continue the boot process as normal.

=== Installing ===
Unfortunately we can't completely rely on the normal installation process because Ubuntu by default doesn't use the iBFT to set up an iSCSI LUN before the installer searches for block devices to install on. We could get around this by simply switching to a TTY, running <code>iscsistart</code> and relaunching the installer, which then identified the block device and proceeded as normal. Since this entire stuff runs off LUNs on the network we can potentially completely get around the installation by just preparing enough LUNS with the appropriate data.

=== iSCSI targets ===
In the home directory of the ubuntuamdin are in the folder iscsi/ scripts for generating the images in the zfs subvolum pool/iscsi (called: generate_zfs_images.sh). Then the iscsi targets are being created with: create_iscsi_targets.sh
With destroy_all_zfs_images.sh everything can be reverted.
The root filesystem size for each node is 80.8GB.

== UEFI settings ==
Settings applied after resetting the UEFI to default settings:
* ...