If Libvirtd is restarted, libvirt drops active domains lose network interface info.

[Steven Wagner]

IMPACT: If libvirt daemon is restarted, it forgets about domain interfaces (breaking domifstat)
RESOLUTION: Check that the domain is inactive before wiping knowledge of it.
PATCH: Based on upstream commit, attached comment #6.
TEST CASE: (Based on Fedora bug report)

{{{
1. Start QEMU/KVM guest with: sudo virsh start vm01
2. restart libvirtd: sudo restart libvrtd
3. Get statistics with sudo virsh domifstat vm01 vnet0

Actual results:
$ sudo virsh domifstat vm01 vnet0
error: Failed to get interface stats vm01 vnet0
error: invalid argument in invalid path, 'vnet0' is not a known interface

Expected results:
$ sudo virsh domifstat vm01 vnet0
vnet0 rx_bytes 83980
vnet0 rx_packets 1612
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 3546
vnet0 tx_packets 15
vnet0 tx_errs 0
vnet0 tx_drop 0

}}}

REGRESSION POTENTIAL: This would seem to be low, the patch adds an extra conditional statement before wiping the domain network interface knowledge. The patch has been in upstream Git since 4/05/2010, and has had at least one upstream release and is also in Maverick.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: libvirt-bin 0.7.5-5ubuntu27.6
ProcVersionSignature: Ubuntu 2.6.32-23.37-server 2.6.32.15+drm33.5
Uname: Linux 2.6.32-23-server x86_64
Architecture: amd64
Date: Thu Oct 28 13:40:36 2010
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: libvirt

[M (m-redhat-bugs)]

Description of problem:
If I have QEMU/KVM guest started in libvirtd and then restart libvirtd, libvirtd no longer knows mapping between guests and tun devices on host and quering virDomainInterfaceStats do not work.

Version-Release number of selected component (if applicable):
libvirt version 0.7.1 (Fedora 12 rpm: libvirt-0.7.1-15.fc12.x86_64)

also tested with Centos 5.4 and libvirt 0.6.3

How reproducible:
every time

Steps to Reproduce:
1. Start QEMU/KVM guest with: virsh start vm01
2. restart libvirtd: /sbin/service libvrtd restart
3. Get statistics with virsh domifstat vm01 vnet0

Actual results:
[root@kvm01 ~]# virsh domifstat vm01 vnet0
error: Failed to get interface stats vm01 vnet0
error: invalid argument in invalid path, 'vnet0' is not a known interface

Expected results:
[root@kvm01 ~]# virsh domifstat vm01 vnet0
vnet0 rx_bytes 83980
vnet0 rx_packets 1612
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 3546
vnet0 tx_packets 15
vnet0 tx_errs 0
vnet0 tx_drop 0

Additional info:
I found out that after libvirtd forgets mapping for virtual interfaces. Here are the output from virsh dumpxml vm01 before restart:
    <interface type='network'>
      <mac address='52:54:00:28:59:5c'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
    </interface>

and after restart of libvirtd:
    <interface type='network'>
      <mac address='52:54:00:28:59:5c'/>
      <source network='default'/>
      <model type='virtio'/>
    </interface>

the <target dev='vnet0'/> is missing.

[Cole (cole-redhat-bugs)]

Thanks for the report, sounds like a real bug.

[Fedora (fedora-redhat-bugs)]

libvirt-0.8.2-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/libvirt-0.8.2-1.fc13

[Fedora (fedora-redhat-bugs)]

libvirt-0.8.2-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/libvirt-0.8.2-1.fc12

[Fedora (fedora-redhat-bugs)]

libvirt-0.8.2-1.fc12 has been pushed to the Fedora 12 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update libvirt'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/libvirt-0.8.2-1.fc12

[Cole (cole-redhat-bugs)]

Danpb, has this really been fixed? I don't remember an upstream patch addressing it.

[Daniel (daniel-redhat-bugs)]

Yep, in this patch:

commit 362bc09a4d9441f9abd14956546777f86d8b9d79
Author: Jiri Denemark <email address hidden>
Date: Tue May 4 15:16:29 2010 +0200

    Don't wipe generated iface target in active domains

    Wipe generated interface target only when reading configuration of
    inactive domains.

[Fedora (fedora-redhat-bugs)]

libvirt-0.8.2-1.fc12 has been pushed to the Fedora 12 stable repository. If problems still persist, please make note of it in this bug report.

[Steven Wagner (stevenwagner)]

Mathias - This is a regression from behavior that was dependable in hardy LTS, and now fails in lucid LTS. Being able to retrieve network interface transfer totals is an essential feature for libvirt.

[Steven Wagner (stevenwagner)]

I filed a bug with the redhat guys, and they gave me the patch info. Need help applying it.

https://bugzilla.redhat.com/show_bug.cgi?id=650543

--- Comment #1 from Daniel Berrange <email address hidden> 2010-11-10 18:15:31 EST ---
Fixed several releases ago by this commit

commit 362bc09a4d9441f9abd14956546777f86d8b9d79
Author: Jiri Denemark <email address hidden>
Date: Tue May 4 15:16:29 2010 +0200

   Don't wipe generated iface target in active domains

   Wipe generated interface target only when reading configuration of
   inactive domains.

[Steven Wagner (stevenwagner)]

Any assistance getting this patch applied? This will fix broken functionality in the LTS release.

[Dave Walker (davewalker)]

Previous comment is a debdiff for lucid-proposed.

Can those affected, please confirm this fix from my PPA by adding the following to /etc/apt/sources.lst:
deb http://ppa.launchpad.net/davewalker/libvirt/ubuntu lucid main

Thanks!

[Steven Wagner (stevenwagner)]

I installed the package Dave. So far so good, ill let it run for a while. Thank you.

[Steven Wagner (stevenwagner)]

Dave - Ive been running the ppa package for almost a week. I can verify the fix works as expected.

[Steven Wagner (stevenwagner)]

how will I be able to tell when this hits lucid-updates ?

[Dave Walker (davewalker)]

@Steven, hopefully it should soon hit the -proposed archive which it's currently in the queue for. Then verification needs to be undertaken on that package (your help there would be appreciated!). Once that is done, it should land in -updates shortly afterwards. Until then, it makes sense to keep using my PPA package (which i've intentionality left the version number such that when this does hit -updates you'll upgrade automatically).

[Steve Langasek (vorlon)]

Accepted into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Imre Gergely (cemc)]

Confirmed.

Before the fix:

gimre@voy:~$ virsh start utest-lls32
Domain utest-lls32 started
gimre@voy:~$ sudo restart libvirt-bin
libvirt-bin start/running, process 8683
gimre@voy:~$ sudo virsh domifstat utest-lls32 vnet0
error: Failed to get interface stats utest-lls32 vnet0
error: invalid argument in invalid path, 'vnet0' is not a known interface

After the fix:

gimre@voy:~$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.7.5-5ubuntu27.8
  Candidate: 0.7.5-5ubuntu27.8
  Version table:
 *** 0.7.5-5ubuntu27.8 0
        500 http://archive.ubuntu.com/ubuntu/ lucid-proposed/main Packages
        100 /var/lib/dpkg/status
     0.7.5-5ubuntu27.7 0
        500 http://ro.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        500 http://ro.archive.ubuntu.com/ubuntu/ lucid-security/main Packages
     0.7.5-5ubuntu27 0
        500 http://ro.archive.ubuntu.com/ubuntu/ lucid/main Packages

gimre@voy:~$ virsh start utest-lls32
Domain utest-lls32 started

gimre@voy:~$ sudo virsh domifstat utest-lls32 vnet0
vnet0 rx_bytes 3294
vnet0 rx_packets 42
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 1912
vnet0 tx_packets 18
vnet0 tx_errs 0
vnet0 tx_drop 0

gimre@voy:~$ sudo restart libvirt-bin
libvirt-bin start/running, process 9232

gimre@voy:~$ sudo virsh domifstat utest-lls32 vnet0
vnet0 rx_bytes 3906
vnet0 rx_packets 54
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 1912
vnet0 tx_packets 18
vnet0 tx_errs 0
vnet0 tx_drop 0

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 0.7.5-5ubuntu27.8

---------------
libvirt (0.7.5-5ubuntu27.8) lucid-proposed; urgency=low

  * debian/patches/9903-lp668042.patch: When libvirtd is restarted, do not
    remove network interface details from active domains. Patched based on
    upstream git commit. (LP: #668042)
 -- Dave Walker (Daviey) <email address hidden> Thu, 25 Nov 2010 02:41:24 +0000