| 2014-08-28 18:40:02 |
Thiago Martins |
bug |
|
|
added bug |
| 2014-08-28 18:40:21 |
Thiago Martins |
bug |
|
|
added subscriber James Page |
| 2014-08-28 20:21:41 |
Thiago Martins |
description |
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0 Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual tagged network within itself. QEmu 2.X guest does not route traffic when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as "firewalls / routers", and it have an easy to reproduce, connectivity problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic) with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them, the QEmu Hypervisors dedicated to host "guest acting as routers / firewalls", like a "borger gateway" for example, that it does not work as expected.
One interesting thing to note is that, this BUG appear only, and only at, the QEmu Hypervisors dedicated to host guests that are used as `router / firewalls` (as I said above), others QEmu Hypervisors of my network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets only when these packets are originated from within the Hypervisor itself, I mean, packets from both host and other guests (not the router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a "border hypervisor". So, this proves that there is something wrong with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-utils`, same bad results. So, don't waste your time trying `bridge-utils` (optional steps while reproducing it), you can keep OVS bridges from original design.
I think that I'm using the best pratices to build this environment, as follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your (virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6 with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest) /etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1` shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with `bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`, instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2', 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the host `qemu-host-1` cannot browse the web again (`apt-get update` will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
The presented workaround have one big downside, while it allows us to keep using QEmu (and KSM), it requires a complete reconfiguration of the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create a fake bridge, a new VirtIO NIC for your guest (this might add a bit of overhead for your hypervisor as a whole, I'm not sure), plus a lot of extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't the best way to build hypervisors - not the best practice), this is just a real workaround that allows you to keep using QEmu (and benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
4- Details: Working as expected when with QEmu but, without tagging the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are originated at its host, neighter from others guests hosted at the same hypervisor. Making it impossible to create a virtual network within a hypervisor.
Best Regards,
Thiago Martins |
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0 Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual tagged network within itself. QEmu 2.X guest does not route traffic when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as "firewalls / routers", and it have an easy to reproduce, connectivity problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic) with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them, the QEmu Hypervisors dedicated to host "guest acting as routers / firewalls", like a "borger gateway" for example, that it does not work as expected.
One interesting thing to note is that, this BUG appear only, and only at, the QEmu Hypervisors dedicated to host guests that are used as `router / firewalls` (as I said above), others QEmu Hypervisors of my network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets only when these packets are originated from within the Hypervisor itself, I mean, packets from both host and other guests (not the router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a "border hypervisor". So, this proves that there is something wrong with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-utils`, same bad results. So, don't waste your time trying `bridge-utils` (optional steps while reproducing it), you can keep OVS bridges from original design.
I think that I'm using the best pratices to build this environment, as follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your (virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6 with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest) /etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1` shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with `bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`, instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2', 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the host `qemu-host-1` cannot browse the web again (`apt-get update` will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
The presented workaround have one big downside, while it allows us to keep using QEmu (and KSM), it requires a complete reconfiguration of the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create a fake bridge, a new VirtIO NIC for your guest (this might add a bit of overhead for your hypervisor as a whole, I'm not sure), plus a lot of extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't the best way to build hypervisors - not the best practice), this is just a real workaround that allows you to keep using QEmu (and benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml, vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are originated at its host, neighter from others guests hosted at the same hypervisor. Making it impossible to create a virtual network within a hypervisor.
Best Regards,
Thiago Martins |
|
| 2014-08-28 20:21:57 |
Thiago Martins |
tags |
|
qemu tag vlan |
|
| 2014-08-28 22:38:11 |
Thiago Martins |
description |
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0 Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual tagged network within itself. QEmu 2.X guest does not route traffic when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as "firewalls / routers", and it have an easy to reproduce, connectivity problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic) with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them, the QEmu Hypervisors dedicated to host "guest acting as routers / firewalls", like a "borger gateway" for example, that it does not work as expected.
One interesting thing to note is that, this BUG appear only, and only at, the QEmu Hypervisors dedicated to host guests that are used as `router / firewalls` (as I said above), others QEmu Hypervisors of my network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets only when these packets are originated from within the Hypervisor itself, I mean, packets from both host and other guests (not the router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a "border hypervisor". So, this proves that there is something wrong with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-utils`, same bad results. So, don't waste your time trying `bridge-utils` (optional steps while reproducing it), you can keep OVS bridges from original design.
I think that I'm using the best pratices to build this environment, as follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your (virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6 with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest) /etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the second hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1` shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with `bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`, instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2', 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the host `qemu-host-1` cannot browse the web again (`apt-get update` will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
The presented workaround have one big downside, while it allows us to keep using QEmu (and KSM), it requires a complete reconfiguration of the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create a fake bridge, a new VirtIO NIC for your guest (this might add a bit of overhead for your hypervisor as a whole, I'm not sure), plus a lot of extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't the best way to build hypervisors - not the best practice), this is just a real workaround that allows you to keep using QEmu (and benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml, vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are originated at its host, neighter from others guests hosted at the same hypervisor. Making it impossible to create a virtual network within a hypervisor.
Best Regards,
Thiago Martins |
Guys,
Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network. It does not route tagged VLAN packets.
That's it, it is impossible to use Trusty acting as a QEmu 2.0 Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual tagged network within itself. QEmu 2.X guest does not route traffic when with tagged VLANs!
So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as "firewalls / routers", and it have an easy to reproduce, connectivity problem.
This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic) with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
I have this very same setup up and running, on about ~100 physical servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them, the QEmu Hypervisors dedicated to host "guest acting as routers / firewalls", like a "borger gateway" for example, that it does not work as expected.
One interesting thing to note is that, this BUG appear only, and only at, the QEmu Hypervisors dedicated to host guests that are used as `router / firewalls` (as I said above), others QEmu Hypervisors of my network does not suffer from this problem.
Another interesting point is that it fails to route tagged VLAN packets only when these packets are originated from within the Hypervisor itself, I mean, packets from both host and other guests (not the router/firewall guest itself), suffer from this connectivity problem.
As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a "border hypervisor". So, this proves that there is something wrong with QEmu.
I already tested it with both `openvswitch-switch` and with `bridge-utils`, same bad results. So, don't waste your time trying `bridge-utils` (optional steps while reproducing it), you can keep OVS bridges from original design.
I think that I'm using the best pratices to build this environment, as follows...
* Topology *
QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"):
1- Physical machine with 3 NICs;
2- Minimal Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpdump".
- eth0 connected to the Internet - VLAN tag 10;
- eth1 connected to the LAN1 - VLAN tag 100;
- eth2 connected to the LAN2 - VLAN tag 200;
Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables):
1- Virtual Machine with 3 NICs (VirtIO);
2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
3- Packages installed: "aiccu iptables vlan pv-grub-menu".
OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your (virt-)lab network, including the `qemu-host-1` itself.
Steps to reproduce
* Preparing the `qemu-host-1` host:
- Configure the /etc/network/interfaces with:
---
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
auto eth1
iface eth1 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
auto ovsbr1p1
iface ovsbr1p1 inet6 auto
iface ovsbr1p1 inet static
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto eth2
iface eth2 inet manual
up ip link set $IFACE up
down ip link set $IFACE down
---
- Creating the Hypervisor OVS Bridges:
ovs-vsctl add-br ovsbr0
ovs-vsctl add-br ovsbr1
ovs-vsctl add-br ovsbr2
- Attaching the bridges to the NICs:
ovs-vsctl add-port ovsbr0 eth0
ovs-vsctl add-port ovsbr1 eth1
ovs-vsctl add-port ovsbr2 eth2
- Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6):
ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 type=internal
ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\"
NOTE:
* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6 with SLAAC, so, it remain fixed across host reboots.
- Making Libvirt aware of OVS Bridges:
Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml and ovsbr2.xml):
--- ovsbr0.xml contents:
<network>
<name>ovsbr0</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr1.xml contents:
<network>
<name>ovsbr1</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
</network>
---
--- ovsbr2.xml contents:
<network>
<name>ovsbr2</name>
<forward mode='bridge'/>
<bridge name='ovsbr2'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define ovsbr0.xml
virsh net-define ovsbr1.xml
virsh net-define ovsbr2.xml
virsh net-autostart ovsbr0
virsh net-autostart ovsbr1
virsh net-autostart ovsbr2
virsh net-start ovsbr0
virsh net-start ovsbr1
virsh net-start ovsbr2
- Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine):
1- VM Configuration file (network-only / cutted):
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='ovsbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='ovsbr1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='ovsbr2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
2- Configure "guest-fw-1.domain.com" (the router / firewall guest) /etc/network/interfaces file like this:
---
auto vlan10
iface vlan10 inet static
vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto vlan100
iface vlan100 inet6 static
vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface vlan100 inet static
vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto vlan200
iface vlan200 inet6 static
vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface vlan200 inet static
vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
3- Enable radvd for your LANs:
---
# SERVERS
interface vlan100 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:200:850a::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL domain.com.br { };
};
# DESKTOPS
interface vlan200 {
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
AdvLinkMTU 1500;
AdvDefaultPreference high;
prefix 2001:1291:2de:10::/64 {
DeprecatePrefix on;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
route ::/0 {
RemoveRoute on;
};
RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { };
DNSSL igcorp.com.br { };
};
---
4- HIT TUE BUG!
Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will not work! Ping works... TCP connections doesn't.
The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
Details:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev ovsbr1p1
192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86397sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# NOTE: the first hope is the "guest-fw-1".
# "apt-get update", not okay! *BUG*
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
# While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1` shows:
http://pastebin.com/2BUiNEfQ
---
(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!)
Possible workarounds: is this an OpenvSwitch BUG? Lets try it with `bridge-utils` instead...
* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`, instead of openvswitch-switch.
------------------------
1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch:
- Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
---
auto br0
iface br0 inet manual
bridge_ports eth0
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto br1
iface br1 inet manual
bridge_ports eth1
bridge_maxwait 5
bridge_fd 1
bridge_stp on
auto vlan100
iface vlan100 inet6 auto
vlan_raw_device br1
iface vlan100 inet static
vlan_raw_device br1
address 192.168.1.10
netmask 24
gateway 192.168.1.1
auto br2
iface br2 inet manual
bridge_ports eth2
bridge_maxwait 5
bridge_fd 1
bridge_stp on
---
2- New VM Configuration file (network-only section / cutted), adjusted to make use bridges from `bridge-utils` package:
---
<interface type='bridge'>
<mac address='52:54:00:41:8c:3f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:27:b2:7d'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:ff:35:5c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
* Start `guest-fw-1` as-is:
virsh start guest-fw-1
New try:
---
root@qemu-host-1:~# ip r
default via 192.168.4.1 dev vlan100
192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
root@qemu-host-1:~# ip -6 r | grep vlan100
2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86397sec
fe80::/64 dev vlan100 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 expires 27sec
# ping6 okay...
root@qemu-host-1:~# ping6 google.com -c1
PING google.com(2800:3f0:4001:815::1007) 56 data bytes
64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms
# traceroute6 okay...
root@qemu-host-1:~# traceroute6 google.com
traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets
1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms 0.223 ms
2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 20.738 ms 20.902 ms
3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms
4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms 141.706 ms 21.058 ms
5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 ms 20.837 ms 20.903 ms
6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms 24.464 ms 24.649 ms
7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 ms 24.525 ms 25.273 ms
8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 ms
9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms 36.375 ms
10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms
11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms 30.769 ms
# BUG effect! "apt-get update", not okay!
root@qemu-host-1:~# apt-get update
0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec
# it remains "Waiting for headers" forever...
- So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, falling back to OpenvSwitch as we started.
** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch.
-- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
---
name = "guest-fw-1"
uuid = "17e031c7-1264-4979-8f06-c5e016469474"
bootloader = "pygrub"
memory = 2048
vcpus = 2
vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2', 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ]
disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ]
---
Details - Working as expected when with Xen!! Look:
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)!
But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the host `qemu-host-1` cannot browse the web again (`apt-get update` will not work if its gateway is a QEmu guest).
** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges".
The presented workaround have one big downside, while it allows us to keep using QEmu (and KSM), it requires a complete reconfiguration of the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create a fake bridge, a new VirtIO NIC for your guest (this might add a bit of overhead for your hypervisor as a whole, I'm not sure), plus a lot of extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't the best way to build hypervisors - not the best practice), this is just a real workaround that allows you to keep using QEmu (and benefits from KSM, Libvirt and etc)...
While, when replacing QEmu by Xen, you don't need to change a single line within the guest itself...
So, this network problem lies within the QEmu Virtual Machine!
Doing this workaround:
1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it:
ovs-vsctl add-br vlan10 ovsbr0 10
ovs-vsctl add-br vlan100 ovsbr1 100
ovs-vsctl add-br vlan200 ovsbr2 100
2- Making Libvirt aware of OVS Fake Bridges:
Create 3 files, one for each fake bridge, like this (vlan10.xml, vlan100.xml and vlan200.xml):
--- vlan10.xml contents:
<network>
<name>vlan10</name>
<forward mode='bridge'/>
<bridge name='vlan10'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan100.xml contents:
<network>
<name>vlan100</name>
<forward mode='bridge'/>
<bridge name='vlan100'/>
<virtualport type='openvswitch'/>
</network>
---
--- vlan200.xml contents:
<network>
<name>vlan200</name>
<forward mode='bridge'/>
<bridge name='vlan200'/>
<virtualport type='openvswitch'/>
</network>
---
Run:
virsh net-define vlan10.xml
virsh net-define vlan100.xml
virsh net-define vlan200.xml
virsh net-autostart vlan10
virsh net-autostart vlan100
virsh net-autostart vlan200
virsh net-start vlan10
virsh net-start vlan100
virsh net-start vlan200
3- Reconfigure the `guest-fw-1` to make use of new "fake bridges":
---
<interface type='network'>
<mac address='52:54:00:41:8c:3f'/>
<source network='vlan10'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:27:b2:7d'/>
<source network='vlan100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:ff:35:5c'/>
<source network='vlan200'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
---
4- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
---
auto eth0
iface eth0 inet static
# vlan_raw_device eth0
address 200.2.1.106
netmask 29
gateway 200.2.1.105
dns-nameserver 8.8.8.8
auto eth1
iface eth1 inet6 static
# vlan_raw_device eth1
address 2001:129X:2XX:810X::2
netmask 64
dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888
iface eth1 inet static
# vlan_raw_device eth1
address 192.168.4.1
netmask 24
auto eth2
iface eth2 inet6 static
# vlan_raw_device eth2
address 2001:1291:2de:10::1
netmask 64
iface eth2 inet static
# vlan_raw_device eth2
address 172.16.0.1
netmask 24
---
5- Details: Working as expected when with QEmu but, without tagging the VLAN within the `guest-fw-1` itself.
---
root@qemu-host-1:~# ping6 -c 1 google.com
PING google.com(2800:3f0:4001:815::1002) 56 data bytes
64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms
root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 86394sec
fe80::/64 dev ovsbr1p1 proto kernel metric 256.
default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 expires 24sec
# *BUG dissapeared!*
root@qemu-host-1:~# apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://us.archive.ubuntu.com trusty-proposed InRelease
Ign http://us.archive.ubuntu.com trusty-updates InRelease
Ign http://us.archive.ubuntu.com trusty-backports InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
........................
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,011 kB in 19s (50.7 kB/s)
Reading package lists... Done
---
Conclusion:
A QEmu guest router does not route tagged VLAN packages that are originated at its host, neighter from others guests hosted at the same hypervisor. Making it impossible to create a virtual network within a hypervisor.
Best Regards,
Thiago Martins |
|
| 2015-01-12 22:53:35 |
Thiago Martins |
affects |
qemu |
linux |
|
| 2015-01-12 22:53:35 |
Thiago Martins |
linux: status |
New |
Confirmed |
|
| 2015-01-12 22:54:22 |
Thiago Martins |
summary |
QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself. |
Linux (3.13 to 3.16) as a Router under QEmu +2, does not route VLAN tagged packets, that are originated within the Hypervisor itself. |
|
| 2015-09-28 04:43:09 |
Thiago Martins |
linux: assignee |
|
Thiago Martins (martinx) |
|
| 2015-09-28 04:43:41 |
Thiago Martins |
linux: status |
Confirmed |
Fix Released |
|
| 2015-09-28 04:43:41 |
Thiago Martins |
linux: assignee |
Thiago Martins (martinx) |
|
|
| 2015-09-28 04:43:55 |
Thiago Martins |
removed subscriber James Page |
|
|
|
