KVM images losing connectivity w/bridged network

[Frank Groeneveld]

Binary package hint: qemu-kvm

I'm opening a new bug as per Serge Hallyn's request in bug #584048.

I experience problems with one of my KVM virtual machines. I have a host running Hardy LTS with the KVM backport ppa (1:84+dfsg-0ubuntu12.1~rc5ppa). The host is using a bridged network with 2 eth's (only 1 is physically connected) and 2 tap devices (for both of my VMs). The virtual machine that works without a problem is running Hardy LTS as well. The machine with problems is running Lucid LTS.

The machines are started as:

screen -t production kvm -m 2048m -nographic -drive file=production.raw,if=virtio,boot=on -net nic,macaddr=52:54:00:12:34:xx,model=virtio -net tap,ifname=tap0,script=no,downscript=no -smp 2

screen -t development kvm -m 1024m -nographic -drive file=development.raw,if=virtio,boot=on -net nic,macaddr=52:54:00:12:34:yy,model=virtio -net tap,ifname=tap1,script=no,downscript=no

Development is the machine giving me problems. After a while (less than 24 hours) of no network activity, the machine loses it's connectivity. We're currently keeping it "running" by sending a ping request every minute and this seems to work, but the network seems pretty slow sometimes.

[Serge Hallyn (serge-hallyn)]

Thanks, Frank, for opening this bug. To help me try to reproduce it, could
you give me the output of the following while the VMs are running:

 brctl show
 ifconfig -a
 iptables -L
 netstat -nr
 /etc/network/interfaces

[Frank Groeneveld (frankgroeneveld)]

Hello Serge,

Here are the requested outputs:

$ brctl show
ridge name bridge id STP enabled interfaces
br0 8000.002219921724 no eth0
       eth1
       tap0
       tap1

$ ifconfig -a
br0 Link encap:Ethernet HWaddr xx:xx:xx:xx:17:24
          inet addr:xx.xx.xx.xx Bcast:xx.xx.xx.xx Mask:255.255.255.0
          inet6 addr: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:603386007 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2296279 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:28072306331 (26.1 GB) TX bytes:391354428 (373.2 MB)

eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:17:24
          inet6 addr: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:761443820 errors:0 dropped:568 overruns:0 frame:0
          TX packets:170054379 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:141300061621 (131.5 GB) TX bytes:160379148403 (149.3 GB)
          Interrupt:16

eth1 Link encap:Ethernet HWaddr xx:xx:xx:xx:17:25
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
          Interrupt:17

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:30 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3296 (3.2 KB) TX bytes:3296 (3.2 KB)

tap0 Link encap:Ethernet HWaddr xx:xx:xx:xx:6f:88
          inet6 addr: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:578969753 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2654211025 errors:0 dropped:3 overruns:9 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:627719982664 (584.6 GB) TX bytes:350948393083 (326.8 GB)

tap1 Link encap:Ethernet HWaddr xx:xx:xx:xx:34:4d
          inet6 addr: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:50380331 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2252027804 errors:0 dropped:117912 overruns:39 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:34513623019 (32.1 GB) TX bytes:152631281449 (142.1 GB)

$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
xx.xx.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
0.0.0.0 xx.xx.xx.4 0.0.0.0 UG 0 0 0 br0

Note, the gateway really is at .4 :)

$ cat /etc/network/interfaces
# This file describes the network interfaces available on you...

[Serge Hallyn (serge-hallyn)]

Thanks, I have a machine up attempting to reproduce. Will check its status tomorrow morning.

In the meantime, two suggestions:

1. I notice eth0 is up with only an ipv6 address. Assuming you're not actually using it for
ipv6, you might try 'ifconfig eth0 down' and see if that helps. Shouldn't make a difference,
but then as I recall from childhood, computers were supposed to not make mistakes.

2. The kvm support in hardy was a technology preview. We can't realistically backport
newer kvm to hardy than what you have because of kernel implications. So if you are
using hardy because it is an LTS, is there any chance of upgrading to the Lucid 10.04.1
LTS release? It's not as intellectually satisfying of a solution, but perhaps a time-saving one.

[Frank Groeneveld (frankgroeneveld)]

> Thanks, I have a machine up attempting to reproduce.  Will check its
> status tomorrow morning.
Great!

> 1. I notice eth0 is up with only an ipv6 address.  Assuming you're not actually using it for
> ipv6, you might try 'ifconfig eth0 down' and see if that helps.  Shouldn't make a difference,
> but then as I recall from childhood, computers were supposed to not make mistakes.

Yes, there is no cable in that ethernetport. Are you really 1000%
sure? Because if I bring it down and my bridge stops, I will have to
travel 150km to go and fix it :)

> 2. The kvm support in hardy was a technology preview.  We can't realistically backport
> newer kvm to hardy than what you have because of kernel implications.  So if you are
> using hardy because it is an LTS, is there any chance of upgrading to the Lucid 10.04.1
> LTS release?  It's not as intellectually satisfying of a solution, but perhaps a time-saving one.

I know, I'm using an LTS release because of the support etc. I'm
planning to upgrade, but I was first testing with this VM. It might
give me problems with the production VM if I upgrade the host system,
and I don't want that (neither do my clients :P). So I am planning to
upgrade, but currently don't dare to do it. In a few weeks, my server
will need to be moved to a new datacenter, so I might then do the
upgrade (easier to debug etc when you're sitting next to it).

>
> --
> KVM images losing connectivity w/bridged network
> https://bugs.launchpad.net/bugs/626662
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Serge Hallyn (serge-hallyn)]

Quoting Frank Groeneveld (<email address hidden>):
> > 1. I notice eth0 is up with only an ipv6 address.  Assuming you're not actually using it for
> > ipv6, you might try 'ifconfig eth0 down' and see if that helps.  Shouldn't make a difference,
> > but then as I recall from childhood, computers were supposed to not make mistakes.
>
> Yes, there is no cable in that ethernetport. Are you really 1000%
> sure? Because if I bring it down and my bridge stops, I will have to
> travel 150km to go and fix it :)

In that case, given how unlikely it is to solve this problem,
and given how badly technology is treating me this week, let's
not.

[Frank Groeneveld (frankgroeneveld)]

Lol, okay :)

Frank

Op 31 aug 2010 21:35 schreef "Serge Hallyn" <email address hidden>:

Quoting Frank Groeneveld (<email address hidden>):
> > 1. I notice eth0 is up with only an ipv6 a...
In that case, given how unlikely it is to solve this problem,
and given how badly technology is treating me this week, let's
not.

--

KVM images losing connectivity w/bridged network
https://bugs.launchpad.net/bugs/626662
You received...

[Serge Hallyn (serge-hallyn)]

This morning the VM and host both still had networking. As I can't reproduce this, and
as you have a working workaround (cheezy periodic ping) and a new LTS to try, I'm
afraid I'm going to have to mark this as wontfix. I'd really like to get to the bottom of it,
so if you happen to find new helpful info, please feel free to comment here.

[Serge Hallyn (serge-hallyn)]

(marking wontfix, with a sense of shame, mostly because kvm is a tech preview in hardy)

[Frank Groeneveld (frankgroeneveld)]

Well, the workaround only gives me connectivity. Performance can be
crap at some times. Thanks for you time, I hope the upgrade will solve
my problems (and not make them worse, production VM should stay
available normally).