NetworkManager, libvirtd & VMs on a routed network using a central DHCP server
I recently upgraded my main workstation to Fedora 16 and decided to change the original bridged setup I had for my Virtual Machines to a routed one. This way I could use the default NetworkManager service and not deviate too much from a default installation. Since it is not too obvious how to set this up I thought I share it for the benefit of others looking to do the same.
Background
The default libvirt network setting in Fedora is NAT. That is fine if you do not need to access multiple VMs from external hosts. If you do need to access your VMs from external hosts or even offer services from those VMs to the Internet then you need to setup a routed network environment (or a bridged one but that one is well documented and requires you to turn off NetworkManager which I don’t want). Also being able to use the central DHCP server for the VMs make life a whole lot easier. So a routed network it is and here are the steps.
Please note that any command prepended with “#” means that you need to execute it as root and any command prepended with “$” can be executed as a regular user.
1. Create the new virtual network device file
First create the new virtual network device file. I called it vmr (r = routed) but you can name it anything you like. Just make sure the bridge name and IP address you choose are not already used.
$ cat > vmr.xml <<EOF <network> <name>vmr</name> <forward mode='route'/> <bridge name='virbr1' /> <ip address='192.168.198.1' netmask='255.255.255.0'> </ip> </network> EOF
2. Add the network to the libvirt config
# virsh net-define vmr.xml
You can verify that the network was indeed added:
# ls -l /etc/libvirt/qemu/networks/ total 12 drwx------. 2 root root 4096 Nov 27 03:14 autostart -rw-r--r--. 1 root root 317 Nov 25 07:35 default.xml -rw-------. 1 root root 490 Nov 28 18:34 vmr.xml
3. Start the new virtual network device
Prerequisite: the libvirtd service must already be running. You can check if it’s running with:
# service libvirtd status
If libvirtd is not yet started then start it with:
# service libvirtd start
Start the vmr device:
# virsh net-start vmr Network vmr started
Let’s check if the vmr device was properly started:
# virsh net-list --all Name State Autostart ----------------------------------------- default active yes vmr active no
Yes it was. Now also to enable Autostart:
# virsh net-autostart vmr Network vmr marked as autostarted
And check if the vmr device will Autostart:
# virsh net-list --all Name State Autostart ----------------------------------------- default active yes vmr active yes
4. Enable ip_forward
For the new routed network to actually work you need to make sure that ip_forward is enabled so check if ip_forward is enabled:
# sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1
So here ip_forward is enabled (“1″). If ip_forward was not enabled then it would have said:
net.ipv4.ip_forward = 0
In that case (it’s not enabled) you need to enable it with:
# sysctl -w net.ipv4.ip_forward=1 # sed -i -e 's/net.ipv4.ip_forward = 0/net.ipv4.ip_forward = 1/' /etc/sysctl.conf
5. Install a DHCP forwarder
For DHCP to work you need to install a DHCP forwarder. The reason is that DHCP broadcasts do not traverse routed networks. Fortunately there is a simple solution:
# yum install dhcp-forwarder
Now put the proper values in the DHCP forwarder configuration:
# vi /etc/dhcp-fwd.conf And add the proper values: if "device_1" true true true if "device_2" true true true server bcast "device_1"
Where “device_1″ is your main network device and “device_2″ is the virtual network device which you used in step 1. In my case my main network device is p21p1 and the virtual network device is virbr1.
Start the DHCP forwarder service:
# systemctl start dhcp-forwarder.service
Make sure the DHCP forwarder service starts when your workstation starts:
# systemctl enable dhcp-forwarder.service
6. Check the firewall rules
Check if libvirt has created the proper firewall rules for our new virtual network device.
# iptables -v -n -L --line-numbers Chain INPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT udp -- virbr1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 2 0 0 ACCEPT tcp -- virbr1 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 3 24 8094 ACCEPT udp -- virbr1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 4 0 0 ACCEPT tcp -- virbr1 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67 5 10242 5615K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 6 2 168 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 7 1 58 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 8 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 9 140 18978 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 39 11254 ACCEPT all -- * virbr1 0.0.0.0/0 192.168.198.0/24 2 66 4686 ACCEPT all -- virbr1 * 192.168.198.0/24 0.0.0.0/0 3 0 0 ACCEPT all -- virbr1 virbr1 0.0.0.0/0 0.0.0.0/0 4 0 0 REJECT all -- * virbr1 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 5 0 0 REJECT all -- virbr1 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 6 4 304 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 804 packets, 218K bytes) num pkts bytes target prot opt in out source destination
That looks good. Libvirtd has created all the required firewall rules for the virtual network device virbr1.
7. Fix the firewall
Unfortunately we are not there yet as the firewall rules block our DHCP responses. And that is cause by rule number #9 in the INPUT chain:
9 140 18978 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
This rule will cause any answer from the DHCP server to be blocked. So we need to fix this by inserting a firewall rule which makes sure valid traffic from our local network on “device_1″ (in my test setup p21p1 with network 10.0.0.0/24) destined for our VM network on “device_2″ (in my test setup virbr1 with network 192.168.198.0/24) is accepted. And it helps if this rule is automagically loaded when you reboot or reload iptables.
First, loose the libvirt iptables rules:
# service iptables restart
Yes restarting the iptables service will make the VM firewall rules created by libvirtd disappear.
Next, add the iptables rule we need. Make sure you replace “device_1″ with your main network interface, “net_1″ with the network of your main network interface and “net_2″ with the network you choose in step 1:
# iptables -I INPUT `iptables -v -L --line-numbers | grep -m1 "REJECT" | cut -d" " -f1` -i device_1 -s net_1/24 -d net_2/24 -j ACCEPT
In my test setup this is the rule with the proper interface and networks:
# iptables -I INPUT `iptables -v -L --line-numbers | grep -m1 "REJECT" | cut -d" " -f1` -i p21p1 -s 10.0.0.0/24 -d 192.168.198.0/24 -j ACCEPT
Now backup any old iptables rule sets you might have:
# mv /etc/sysconfig/iptables /etc/sysconfig/iptables.old
And save the current ruleset:
# iptables-save > /etc/sysconfig/iptables
The iptables rule set including the custom rule above should now automagically load when the workstation starts or when you reload the iptables service.
8. Add routes
Finally don’t forget to add the proper routes on any other boxes or your ADSL router that point to your virtual VM network if you want to allow access to your VMs from for example other hosts on your network or from the Internet.
An example of the route you would add on another Linux workstation or server could be:
# route add -net "net_2" netmask 255.255.255.0 gw "ip_of_device_1" dev "device_1"
In my test setup this is the rule with the proper interface and networks:
# route add -net 192.168.198.0 netmask 255.255.255.0 gw 10.0.0.135 dev p21p1
9. Do not forget…
To make sure you use a properly configured firewall on your VMs if you choose to expose them to the Internet!
To send a SIGHUP to libvirtd every time you reload iptables or else the libvirtd iptables rule set is lost and you will no longer be able to reach your VMs:
# pkill -SIGHUP libvirtd
10. The End
That’s it. When your VMs start up they will be able to get an IP address from a central DHCP server (assuming the VM is configured for DHCP off course) and you will be able to access your VMs from your local network and even the Internet of you choose to do so. Comments and improvements always welcome.