Skip to content

Rescue Mode

Rescue mode is a recovery mechanism for nodes that are no longer reachable or fail to boot. It works by booting the node into a minimal Linux environment over the network, bypassing the installed operating system entirely. This gives you SSH access to the node so you can inspect and repair the underlying disks — for example, to fix a broken bootloader, recover from a bad firewall rule, or restore configuration files. The node retains its IP address in rescue mode, so you can connect using the same address as usual, with a temporary password you set before activating rescue mode.

To boot a node into rescue mode, follow these steps:

  1. Go to the node's access page in the panel.
  2. Scroll down to "Rescue Mode", enter an SSH password.
  3. Click "Boot into Rescue Mode".
  4. Wait for the node to boot into rescue mode.

The node will be reachable under the regular IP address, but you will need to use the password you entered in step 2 to log in.

Reset the host key

The node will receive a new host key when it boots into rescue mode. This means that you will see a warning when you try to SSH into the node. You can remove the old host key from your known hosts file by running the following command:

ssh-keygen -R <ip>

Replace <ip> with the IP address of the node.

Login to rescue mode

Use the username rescue and the password you entered in step 2 to log in to the rescue mode via SSH:

ssh rescue@<ip>

Replace <ip> with the IP address of the node.

Use chroot to access the original system

The rescue mode does not mount the original system, but you can use chroot to access it. This allows you to fix the problem that caused the node to be unreachable.

To access the original system, follow these steps:

  1. Find the root partition of the original system. You can list the partitions with the following command:
[root@localhost rescue]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 953.9G  0 disk
|-nvme0n1p1 259:1    0    54M  0 part
|-nvme0n1p2 259:2    0   512M  0 part
|-nvme0n1p3 259:3    0 953.3G  0 part

The root partition is usually the one with the most space. In this case, it is /dev/nvme0n1p3. The boot partition is usually /dev/nvme0n1p2. The EFI partition is usually /dev/nvme0n1p1, the smallest partition.

  1. Mount the root partition to /mnt:
mount /dev/nvme0n1p3 /mnt
mount /dev/nvme0n1p2 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -t devpts pts /mnt/dev/pts
  1. Chroot into the system:
chroot /mnt /bin/bash

You are now in the original system. You can fix the problem that caused the node to be unreachable.

  1. When you are done, exit the chroot:
exit
  1. Go back to the panel and click on "Leave Rescue Mode" to reboot the node into the original system.