How to fix Fedora 16 boot problem when using RAID

If you use mdraid (Linux’ software RAID) and you want to install Fedora 16 with /boot, / and swap each in a RAID1 setup your box may not boot once the installation is finished. First go read bugzilla bug 750794 to understand what seems to be happening and why your box with shiny new Fedora 16 may not boot. Here is what I had to do to fix this problem. When you see the “Congratulations…” message at the end of the installation do not reboot. Instead go to the console (alt-f2) and do the following:

1. add the RAID modules to the grub2 environment

2. generate a new grub2 config

3. install grub2 again on both your RAID1 disks. Replace sda & sdb with your devices!!!

Your box should now properly boot. If it does, enjoy your new Fedora 16 installation!

NetworkManager, libvirtd & VMs on a routed network using a central DHCP server

I recently upgraded my main workstation to Fedora 16 and decided to change the original bridged setup I had for my Virtual Machines to a routed one. This way I could use the default NetworkManager service and not deviate too much from a default installation. Since it is not too obvious how to set this up I thought I share it for the benefit of others looking to do the same.

Background

The default libvirt network setting in Fedora is NAT. That is fine if you do not need to access multiple VMs from external hosts. If you do need to access your VMs from external hosts or even offer services from those VMs to the Internet then you need to setup a routed network environment (or a bridged one but that one is well documented and requires you to turn off NetworkManager which I don’t want). Also being able to use the central DHCP server for the VMs make life a whole lot easier. So a routed network it is and here are the steps.

Please note that any command prepended with “#” means that you need to execute it as root and any command prepended with “$” can be executed as a regular user.

1. Create the new virtual network device file

First create the new virtual network device file. I called it vmr (r = routed) but you can name it anything you like. Just make sure the bridge name and IP address you choose are not already used.

2. Add the network to the libvirt config

You can verify that the network was indeed added:

3. Start the new virtual network device

Prerequisite: the libvirtd service must already be running. You can check if it’s running with:

If libvirtd is not yet started then start it with:

Start the vmr device:

Let’s check if the vmr device was properly started:

Yes it was. Now also to enable Autostart:

And check if the vmr device will Autostart:

4. Enable ip_forward

For the new routed network to actually work you need to make sure that ip_forward is enabled so check if ip_forward is enabled:

So here ip_forward is enabled (“1”). If ip_forward was not enabled then it would have said:

In that case (it’s not enabled) you need to enable it with:

5. Install a DHCP forwarder

For DHCP to work you need to install a DHCP forwarder. The reason is that DHCP broadcasts do not traverse routed networks. Fortunately there is a simple solution:

Now put the proper values in the DHCP forwarder configuration:

Where “device_1” is your main network device and “device_2” is the virtual network device which you used in step 1. In my case my main network device is p21p1 and the virtual network device is virbr1.

Start the DHCP forwarder service:

Make sure the DHCP forwarder service starts when your workstation starts:

6. Check the firewall rules

Check if libvirt has created the proper firewall rules for our new virtual network device.

That looks good. Libvirtd has created all the required firewall rules for the virtual network device virbr1.

7. Fix the firewall

Unfortunately we are not there yet as the firewall rules block our DHCP responses. And that is cause by rule number #9 in the INPUT chain:

This rule will cause any answer from the DHCP server to be blocked. So we need to fix this by inserting a firewall rule which makes sure valid traffic from our local network on “device_1” (in my test setup p21p1 with network 10.0.0.0/24) destined for our VM network on “device_2” (in my test setup virbr1 with network 192.168.198.0/24) is accepted. And it helps if this rule is automagically loaded when you reboot or reload iptables.

First, loose the libvirt iptables rules:

Yes restarting the iptables service will make the VM firewall rules created by libvirtd disappear.

Next, add the iptables rule we need. Make sure you replace “device_1” with your main network interface, “net_1” with the network of your main network interface and “net_2” with the network you choose in step 1:

# iptables -I INPUT iptables -v -L --line-numbers | grep -m1 "REJECT" | cut -d" " -f1 -i device_1 -s net_1/24 -d net_2/24 -j ACCEPT

In my test setup this is the rule with the proper interface and networks:

# iptables -I INPUT iptables -v -L --line-numbers | grep -m1 "REJECT" | cut -d" " -f1 -i p21p1 -s 10.0.0.0/24 -d 192.168.198.0/24 -j ACCEPT

Now backup any old iptables rule sets you might have:

And save the current ruleset:

The iptables rule set including the custom rule above should now automagically load when the workstation starts or when you reload the iptables service.

8. Add routes

Finally don’t forget to add the proper routes on any other boxes or your ADSL router that point to your virtual VM network if you want to allow access to your VMs from for example other hosts on your network or from the Internet.

An example of the route you would add on another Linux workstation or server could be:

In my test setup this is the rule with the proper interface and networks:

9. Do not forget…

To make sure you use a properly configured firewall on your VMs if you choose to expose them to the Internet!

To send a SIGHUP to libvirtd every time you reload iptables or else the libvirtd iptables rule set is lost and you will no longer be able to reach your VMs:

10. The End

That’s it. When your VMs start up they will be able to get an IP address from a central DHCP server (assuming the VM is configured for DHCP off course) and you will be able to access your VMs from your local network and even the Internet of you choose to do so. Comments and improvements always welcome.