Rescue Mode

If a node will no longer boot or is unreachable due a configuration mistake (e.g. firewall) you can recover the node manually by using the rescue mode. This will allow you to access the node via SSH and fix the problem.

The rescue mode is a minimal Linux environment that is booted from the network. It is not a full operating system, but it allows you to access the disks of the node and fix the problem.

To boot a node into rescue mode, follow these steps:

1. Go to the node's access page in the panel.
2. Scroll down to "Rescue Mode", enter an SSH password.
3. Click "Boot into Rescue Mode".
4. Wait for the node to boot into rescue mode.

The node will be reachable under the regular IP address, but you will need to use the password you entered in step 2 to log in.

Reset the host key

The node will receive a new host key when it boots into rescue mode. This means that you will see a warning when you try to SSH into the node. You can remove the old host key from your known hosts file by running the following command:

ssh-keygen -R <ip>

Replace <ip> with the IP address of the node.

Login to rescue mode

Use the username rescue and the password you entered in step 2 to log in to the rescue mode via SSH:

ssh rescue@<ip>

Replace <ip> with the IP address of the node.

Use chroot to access the original system

The rescue mode does not mount the original system, but you can use chroot to access it. This allows you to fix the problem that caused the node to be unreachable.

To access the original system, follow these steps:

1. Find the root partition of the original system. You can list the partitions with the following command:

   [root@localhost rescue]# lsblk
   NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   nvme0n1     259:0    0 953.9G  0 disk
   |-nvme0n1p1 259:1    0    54M  0 part
   |-nvme0n1p2 259:2    0   512M  0 part
   |-nvme0n1p3 259:3    0 953.3G  0 part
   

   The root partition is usually the one with the most space. In this case, it is /dev/nvme0n1p3. The boot partition is usually /dev/nvme0n1p2. The EFI partition is usually /dev/nvme0n1p1, the smallest partition.

2. Mount the root partition to /mnt:

   mount /dev/nvme0n1p3 /mnt
   mount /dev/nvme0n1p2 /mnt/boot
   mount /dev/nvme0n1p1 /mnt/boot/efi
   mount -t proc proc /mnt/proc
   mount -t sysfs sys /mnt/sys
   mount -o bind /dev /mnt/dev
   mount -t devpts pts /mnt/dev/pts
   

3. Chroot into the system:

   chroot /mnt /bin/bash
   

   You are now in the original system. You can fix the problem that caused the node to be unreachable.

4. When you are done, exit the chroot:

   exit
   

5. Go back to the panel and click on "Leave Rescue Mode" to reboot the node into the original system.