Configuring IPMI Serial-over-LAN on Debian 8 (Jessie)

What is IPMI? Why still use a serial console?

While your desktop or laptop at home isn’t all that far away from you, a server in a remote datacenter isn’t usually within reach. In case of issues with the machine or its operating system, it might be handy to have access to its keyboard and mouse inputs and video output to diagnose boot errors, kernel panics and all that sort of thing. On decent servers, a Baseboard Management Controller (BMC) exists, which speaks the IPMI protocol. This allows you to reboot a machine, power it up or down, read out some sensor data, etc. On modern servers, this function is supplemented by a web server which can also serve you the same data through your browser, which makes things a little more accessible. Even better if there’s a way to reach the KVM (Keyboard, Video, Mouse) from there.

This isn’t always the case however. On HP servers, to access this after the OS has booted requires an “iLO Advanced License” which may not be in your budget, or simply not activated on that one older server you’re still using. Older servers, like ones based on Supermicro’s IPMI 2.0 add-in boards simply don’t have this function. Or you may have a BMC which uses a Java applet to show you the KVM, which no longer works with a recent Java version, or even worse, it’s an ActiveX object that only works in IE6!

Have no fear, there’s another nice function that IPMI v2.0 provides: Serial-over-LAN. For a Windows server, that likely doesn’t mean much, but for a Linux server, we can definitely make use of this. I’ll be using Debian in this article, but the same will go for Ubuntu (likely with identical commands and files) and any other Linux distribution (which may use different paths, commands, etc).

After setting this up for one of my home test servers that’s usually powered off, I’ve used it many times to watch its boot process remotely – It’s got 5 RAID controllers installed, so booting takes a while.

Prerequisites

For this set-up I will assume you have configured your IPMI card to be remotely accessible (IP address, gateway, etc), with a user configured. You can try this out fairly easily from a remote machine:

debian:~# ipmitool -H <ip address> -U <username> chassis status
Password:
System Power         : on
Power Overload       : false
(...)

This example is based on Debian “Jessie” 8.0 running on a Supermicro AOC-IPMI20-E (“BMC2”) on a X6DH8 board. It’s old, but set up should be fairly identical on newer machines. This BMC does not have remote KVM access at all, so Serial-over-LAN is the only way to get any access to the OS through the management controller.

On my old (2006!) Supermicro card, I had to upgrade the firmware from “IPMI 2.0 RCMP” to “IMPI 2.0 RCMP+” for Serial-over-LAN to work. This is because that feature was formally introduced in RCMP+ – some BMCs (Tyan, Intel, Supermicro) had custom extensions so some of their IPMI 1.5 controllers could already do it.

Try to connect to the IPMI serial port (using the “LAN Plus” interface, which means IPMI 2.0, as the default “LAN” interface is IPMI 1.5):

debian:~# ipmitool -H <ip address> -U <username> -I lanplus sol activate
Password:
[SOL Session operational.  Use ~? for help]

Unless you already have something configured to speak serial, things will just stay blank there. Exiting that SOL session can be done by typing “~.” like typing “~?” will tell you.

You can find the correct bit rate using the sol info command:

debian:~# ipmitool -H <ip address> -U <username> -I lanplus sol info 1
(...)
Volatile Bit Rate (kbps) : 19.2
Non-Volatile Bit Rate (kbps) : 19.2
(...)

On my old card I can’t seem to change the serial speed; if you can, I do recommend setting it to 115200bps as it’ll very much speed up your boot process. You can try doing this using ipmitool sol set volatile-bit-rate 115.2. If you succeed at changing this, be sure to replace 19200 by the bitrate you’re using in all commands and settings used below.

I used “screen /dev/ttySx 19200” on the server to find the correct serial port that is connected to the BMC – in my case (I think on all Supermicro boards) this was /dev/ttyS1 aka COM2. Just type something into the screen session, and if you see it appear on your remote IPMI Serial-over-LAN, you found the correct port.

Sending your BIOS output to the serial port

This is very likely Supermicro-specific, but setup for your board should be very similar. As we’ve established COM2 was the correct serial port, we enable Remote Access to the BIOS over COM2 (“COM B” in Supermicro BIOS), using 19200 8n1 serial settings.

Advanced Options > Console Redirection

Console Redirection

Make sure to disable “Enable Console Redirection after POST” in the BIOS setup not to conflict with GRUB’s serial terminal code.

Using this setup, you can even control your RAID controller BIOSes or similar. Of course, if this controller has a graphical interface (like LSI/Intel has), that won’t pass over the serial link. Boo!

You can even do a PXE boot, but once the PXE image has booted, it’ll have to do its own serial port output (unless you enable “Enable Console Redirection after POST”).

Sending your GRUB2 output to the serial port

Open /etc/default/grub and change the following settings:

GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=19200 --unit=1 --word=8 --parity=no --stop=1"

Important note: that “serial console” isn’t a text string to tell GRUB2 to use a serial console. They’re 2 distinct words; as contrary to GRUB 0.x, GRUB2 supports 2 simultaneous I/O terminals – “serial” for the serial port, “console” for the machine’s native monitor/keyboard console.

In GRUB_SERIAL_COMMAND, “--unit=1” means ttyS1 aka COM2.

After changing the configuration, be sure to run update-grub – but check the next section first, as there actually is some more work to do.

Sending your Linux kernel output to the serial port

Open /etc/default/grub and modify the GRUB_CMDLINE_LINUX_DEFAULT line:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS1,19200n8"

I would recommend to remove “quiet” to get the full kernel output over your serial port – if you leave it, you won’t see a whole lot of output on your serial port at all.

After changing the configuration, be sure to run update-grub.

Enabling a serial console login on the serial port

Older HOWTOs likely point you at /etc/inittab – a file which is no longer used by systemd (which Debian 8 is using). The good news is, if you’ve enabled GRUB2 to use a serial port, systemd will automatically add a listener on that serial port.

Memtest86+

If you have the memtest86+ package installed, it will automatically add a few entries to your grub menu to run memtest with serial port support. However, if your IPMI SOL port speed is not 115200 and/or your serial port is not ttyS1/COM1, it might be useful to modify /etc/grub.d/20_memtest86. Replace 115200 by your correct bitrate and ttyS0 by the correct serial port. Make sure to run update-grub after editing this file.

Conclusion

If everything went well, you how have a fully serial-controllable system! From booting the BIOS, to the boot loader, over the Linux kernel out right to the terminal login prompt, everything is accessible to you via the IPMI serial console.

Writing informative technical how-to documentation takes time, dedication and knowledge. Should my blog series have helped you in getting things working the way you want them to, or configure certain software step by step, feel free to tip me via PayPal (paypal@powersource.cx) or the Flattr button. Thanks!
  • Jaime Lopez

    Hi. I’m facing a Dell Power Edge 2950 server which runs Xen hypervisor. I’ve configured server to get SoL communication, I added some lines to /etc/default/grub and can see BIOS, GRUB and getty. However I have tested two different configurations and with one of them I’m able to see Xen messages in SoL and with the other one Kernel messages on SoL. I cannot figure out how to get both. Any suggestion?
    thank you in advance!

    • sid3windr

      Hi Jaime,

      I haven’t used Xen in a very long time and do recall having exactly this problem as well (back when using an actual serial console server and not IPMI SoL, but the issue is the same) ! Unfortunately I don’t remember if and how I solved it – I do know my web searches turned up a lot of people having the same issue but without a solution.

      I just took a quick look in my old machine backups looking for clues in the Grub configuration. This was Grub 0.99 on Debian 6 with Xen 3.2, but maybe one of these can give you a hint?


      ## default kernel options
      # kopt=root=/dev/mapper/storage-system ro console=tty0 console=hvc0

      ## Xen hypervisor options to use with the default Xen boot option
      # xenhopt=com1=9600,8n1 console=com1,vga dom0_mem=384M

      ## Xen Linux kernel options to use with the default Xen boot option
      # xenkopt=console=tty0 console=hvc0


      title Xen 3.2-1-amd64 / Debian GNU/Linux, kernel 2.6.26-2-xen-amd64
      root (hd0,0)
      kernel /xen-3.2-1-amd64.gz com1=9600,8n1 console=com1,vga dom0_mem=384M
      module /vmlinuz-2.6.26-2-xen-amd64 root=/dev/mapper/storage-system ro console=tty0 console=hvc0 console=tty0 console=hvc0
      module /initrd.img-2.6.26-2-xen-amd64

      • Jaime Lopez

        Thank you sid3windr. I found a similar configuration in the Xen’s wiki for GRUB and I found consistency with yours. tested the new configuration and get Kernel and Xen messages but the output it’s a kind of “unstable”. I think I would configure to see kernel messages with the proper lines commented to switch to Xen messages.
        Thank you.

        • sid3windr

          Alright, I hope you get some use out of it. I don’t recall instability but I switched to KVM many many moons ago 🙂

          Thanks for reading and following up!