Howto fix CentOS 5 VM not booting with a kernel panic

While I have moved on from CentOS 5 a long time ago, on rare occasions I need to access an oldĀ  CentOS 5 VM. Today was such an occasion and it resulted in the VM not booting with a kernel panic. It took a bit of digging to figure out what was going on and how to fix it so I though I share it to save you some time.

The symptoms

When booting the CentOS 5 VM I saw the following messages:

Note the Trying to resume from label=SWAP-vda3 followed by Unable to access resume device (LABEL=SWAP-vda3). Apparently last time this VM was used (running on a CentOS 5 host) it was perhaps paused or saved which is the reason why it now wants to resume from swap. And something seems wrong with the swap space in the VM. It might be corrupt or using this CentOS 5 VM on a CentOS 7 host instead of it’s old CentOS 5 host requires some changes.

So the swap partition and references to it in /etc/fstab need to be checked and fixed if neccessary. At least that’s obvious.

What’s not so obvious is that when a CentOS 5 VM goes into such a state is that the initrd will be recreated with instructions to resume from swap. So not only do you need to fix the swap space, check /etc/fstab and fix any references if required but you also need to recreate the initrd.

Here are the steps to fix both issues:

Boot VM with rescue mode

Boot the VM and choose F12 during boot so you can select PXE. Once the PXE options are available you don’t boot from local harddisk but instead boot a rescue image. I just used a CentOS 6 rescue entry which I always have available when booting with PXE.

The PXE label config looks like this:

And the rescue.ks kickstart file looks like this:

Once you have booted into rescue mode your CentOS 5 VM harddisks should be automagically mounted and you are presented with the option to start a shell, run a diagnostic or reboot. Select Start shell and activate the chroot:

Quick tip: you can now start the SSH server so you can ssh into the VM and do your work from a terminal instead of the VM console.

Check and fix swap

Let’s see what partitions this VM has:

And let’s see what the label of the swap partition is:

And if that fails, try:

So there are references to SWAP-vda3 (where vda refers to a virtio disk) in a VM that only ever had IDE named devices. That does not seem right. Let’s recreate the swap partition with the proper label:

Check and fix /etc/fstab

Let’s see what’s in /etc/fstab:

Note again the reference to the vda3 (virtio) partition while the label of our swap partition is now SWAP-sda3. So let’s fix the swap line in /etc/fstab so it has the proper label:

Optionally install the latest updates

Before recreating the initrd you can optionally install the latest updates if you have not updated the VM recently and feel adventurous. The reason I mention adventurous is that if you run yum update while booted via a rescue image and accessing the VM via chroot and a new kernel is installed, you will probably see the following error during the kernel installation:

The reason for the error is that when booting via a rescue image and accessing the VM via chroot, /dev/root does not point to the root partition (/dev/sda2 in this VM). Unfortunately the scripts that are run when a new kernel is installed, new-kernel-pkg and grubby, can’t handle that situation gracefully so grub.conf is most likely not updated with the new kernel details. If that is the case you will need to manually add an entry for the new kernel to /boot/grub/grub.conf:

Note that hd(0,0) might not apply to your setup and require a different entry.

Recreate the initrd

Go to the /boot directory and check what the version is of the most recent kernel:

So the version is 2.6.18-371.12.1.el5

Next move the old initrd out of the way:

Now let’s recreate the initrd:

Also check if the latest kernel is the default one that gets booted. There should be an entry default=0 in /boot/grub/grub.conf. The 0 means that the first kernel entry (the one at the top with kernel version 2.6.18-371.12.1) will be used.

Before you reboot

Before you reboot, check the hardware details of the VM. In my case the VM required the Disk bus of the harddisk to be IDE. Check the setting in virtual-manager or in the xml config file of the VM and make sure you have the correct settings.

Reboot

Finally reboot the VM to see if it worked. From the console:

sh-3.2# sync
sh-3.2# exit
bash-4.1# exit

And then select reboot followed by Ok.

Janus WebRTC Gateway on CentOS 6

WebRTC is an exciting innovation that enables Real-Time Communications (both HD audio and video) using just a browser. Last week I looked at Janus – a general puporse WebRTC gateway developed by Meetecho. The demo’s show the power of WebRTC and are definitely worth a look.

Currently the IETF’s latest WebRTC Audio Codec and Processing Requirements draft states that the client is required to support Opus for audio. This means that if you want to stream audio from a WebRTC server to a client, the server will need to be able to do so using the Opus audio codec. And that’s where things get interesting.

Trying to build Janus on CentOS 6.5 x86_64 I bumped into several issues:

1) Janus requires a newer glib than available on CentOS 6
2) Janus requires a newer libnice than available on CentOS 6
3) Janus requires the Opus plugin in GStreamer

Here’s how it went:

Janus requires a newer glib than available for CentOS 6

Since glib is part of CentOS 6 ‘base’ it’s a bad idea to just create a RPM package of a newer glib and replace the existing one. Even if there was no issue with replacing base RPMs it would still require a rebuild of all the RPMs that depend on glib. And that’s a lot of RPMs meaning rebuilding large parts of CentOS. So I looked into creating an RPM of the required glib with the autotools stuff patched to create a non-conflicting namespace. After spending way too much time in the Autotools rabbit hole I decided to just create static glib libs and link Janus against those static libs.

Janus requires a newer libnice than available for CentOS 6

Similar to glib I decided to create static libnice libs and link Janus against those static libs.

Linking static and dynamic libs

With the glib and libnice libs statically compiled I went on to build Janus. While building went well, linking failed consistently. After way too much time staring at linking failures and trying to figure out why it was failing to link, I found out that there is some GCC linker voodoo required if you want to link to both static and dynamic libraries. The simplified version is:

Once I added that and figured out in which order the libs needed to be listed I finally had a Janus executable:

Janus requires the Opus plugin in GStreamer

CentOS 6 has gstreamer-plugins-bad-free version 0.10.19 which does not contain the Opus plugin. So I figured I just rebuild it with the Opus plugin enabled and stick the Opus plugin in a separate package so it does not interfere with gstreamer-plugins-bad-free from ‘base’. That sounded like a good idea until I found out that gstreamer-plugins-bad-free version 0.10.19 did not even contain the Opus plugin. The first version to get the Opus plugin is gstreamer-plugins-bad-free version 0.10.23 which happens to be shipped with CentOS 7. So I decided to take the Opus plugin code from the 0.10.23 release and add it to the 0.10.19 release resulting in one big patch adding everything including the autotools part. Unfortunately the GStreamer developers decided to introduce a new API with the 0.10.23 release. Since that code is not present in the 0.10.19 release the Opus plugin can not be compiled unless that API is backported which is not part of my skillset.

Close but no sigar

In hindsight I should have looked just deep enough at each issue to see if there were any deal breakers. The lack of a Gstreamer Opus plugin on EL6 means that at least Janus can not be used on EL6. And since Opus is an audio requirement for WebRTC there are probably other WebRTC projects that are challenging to deploy on EL6. And that’s not even taking into account the required VP8 and other video codecs. WebRTC is shiny new technology with shiny new requirements. If you are planning a WebRTC Proof of Concept, investigate all possible issues and jump straight to EL7.

Retry on CentOS 7

Once I have access to a CentOS 7 build environment I will revisit Janus and share the results.

Howto add search domains to resolv.conf using dhcpd

If you want to be able to search multiple domain names on a desktop that uses DHCP then you need to make sure those domain names get added to /etc/resolv.conf. The way to do that is to add the DHCP option domain-search specifying the list of domain names to /etc/dhcp/dhcpd.conf:

Please note that the search list is limited to six domains with a total of 256 characters.