Sometimes your linux machine locks up, and you need to reboot it. You could just hit the reset button, but what if there’s something that’s still running, writing data, and turning it off abruptly could cause data loss.

In the event that you have a server that is locked up, but is not a kernel panic, you can use the SysRq keys to reboot the system. They are a keys that will kill processes and unmount the hard disk in the even that a system locks up, but the kernel is still running.

Via the Keyboard

To use them, use the Alt + SysRq plus whichever key matches the action you want to do. Here are the keys:

Here’s some of the main ones:

  • r – Raw mode. This puts the keyboard into ‘raw’ mode, taking it away from everything else.
  • e – End or rather send the terminate signal to all processes, allowing them to end gracefully.
  • i – Immediately send the kill signal to all processes.
  • s – Sync the disk caches and flush them.
  • u – Unmount the drives read-write, making them read-only.
  • b – Boot the machine (well, it actually reboots.)

And that’s actually the order to use to reboot a locked up machine. There’s a mnemonic “Raising Elephants Is So Utterly Boring” which can help to remember it. As in, on a locked up machine you can hold Alt + SysRq and then (one at a time while holding the other two) hit r e i s u b.

Some other ones that may be helpful are

  • n – ‘Niceify’ (or renice) all high and realtime process. Useful if you have a runaway high priority process running.
  • f – Foot and kill any OOM process that’s hogging all the memory.
  • o – Off the computer.

Some other keys can be found (here)[https://en.wikipedia.org/wiki/Magic_SysRq_key].

Via the Command line

If you need to make one of these happen over the command line, or ssh, you can echo it to the /proc/sysrq-trigger such as:

echo s > /proc/sysrq-trigger

This will sync the disks. To run a different one, substitute the s with the key you want to send.

Making Sure it’s Enabled

Now the catch is that you have to have the Magic SysRq enabled. If you want to see if SysRq is active, run:

cat /proc/sys/kernel/sysrq

And verify it’s 1 or higher. If it’s 0, run echo 1 > /proc/sys/kernel/sysrq

You can add this to your sysctl by making a file, such as /etc/sysctl.d/99-sysctl.conf

sysctl kernel.sysrq=1

Note that the systemd init may not play nicely with Magic SysRq. One bug filed against it said that systemd may try to restart services, after Alt+SysRq+e/i is sent, and that they will not fix that.