Nova tries to re-define an existing nwfilter with the same name but different uuid

Bug #1360119 reported by David Hill
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Daniel Berrange
Icehouse
Fix Released
Undecided
Unassigned

Bug Description

Hello,

    I have successfully compiled libvirt 1.2.7 and qemu 2.1.0 but had some troubles with nova-compute. It appears like libvirt is throwing back an error if a nwfilter is already present.

Here is my debug log:
2014-08-22 08:22:25.032 15354 DEBUG nova.virt.libvirt.firewall [req-0959ec86-3939-4e38-9505-48494b44a9fa f1d21892f9a0413c9437b6771e4290ce 9cad53a0432d4164837b8c0b35d91307] nwfilterDefineXML may have failed with (operation failed: filter 'nova-nodhcp' already exists with uuid 59970732-ca52-4521-ba0c-d001049d8460)! _define_filter /usr/lib/python2.6/site-packages/nova/virt/libvirt/firewall.py:239
2014-08-22 08:22:25.033 15354 DEBUG nova.virt.libvirt.firewall [req-0959ec86-3939-4e38-9505-48494b44a9fa f1d21892f9a0413c9437b6771e4290ce 9cad53a0432d4164837b8c0b35d91307] nwfilterDefineXML may have failed with (operation failed: filter 'nova-base' already exists with uuid b5aa80ad-ea4a-4633-84ac-442c9270a143)! _define_filter /usr/lib/python2.6/site-packages/nova/virt/libvirt/firewall.py:239
2014-08-22 08:22:25.034 15354 DEBUG nova.virt.libvirt.firewall [req-0959ec86-3939-4e38-9505-48494b44a9fa f1d21892f9a0413c9437b6771e4290ce 9cad53a0432d4164837b8c0b35d91307] nwfilterDefineXML may have failed with (operation failed: filter 'nova-vpn' already exists with uuid b61eb708-a9a5-4a16-8787-cdc58310babc)! _define_filter /usr/lib/python2.6/site-packages/nova/virt/libvirt/firewall.py:239

Here is the original function:
    def _define_filter(self, xml):
        if callable(xml):
            xml = xml()
        self._conn.nwfilterDefineXML(xml)

And here is the "patched" function":
    def _define_filter(self, xml):
        if callable(xml):
            xml = xml()
        try:
            self._conn.nwfilterDefineXML(xml)
        except Exception, e:
            LOG.debug(_('nwfilterDefineXML may have failed with (%s)!'), e)

I'm not a python expert but I think that patch could be adapted to raise an error ONLY if the nwfilter rule doesn't already exist.

Dave

Revision history for this message
Daniel Berrange (berrange) wrote :

"operation failed: filter 'nova-nodhcp' already exists with uuid 59970732-ca52-4521-ba0c-d001049d846"

Libvirt aims to ensure uniqueness of name + uuid. So if you have a filter already existing with name FOO and UUID of XXXXX, and you try to define a new one with name FOO but UUID of YYYY it considers that a mistake.. The fix is to ensure you always use the original UUID when trying to replace the config of the existing filter.

summary: - Nova-compute bug with libvirt 1.2.7
+ Nova tries to re-define an existing nwfilter with the same name but
+ different uuid
Revision history for this message
Daniel Berrange (berrange) wrote :

Specifically this bug means Nova should be changed to preserve the existing UUID when re-defining nw filters avoiding the raising of the error

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Daniel Berrange (berrange) wrote :

Changed the priority to high, since this causes the nova-compute process to abort during startup if any VMs are running on the host in question. I'm working on a fix to preserve the uuid.

Changed in nova:
importance: Low → High
assignee: nobody → Daniel Berrange (berrange)
Revision history for this message
Daniel Berrange (berrange) wrote :

FYI, Nova has always been broken in this regard, but libvirt was buggy and did not enforce uuid,name stability for nwfilters until release 1.2.7

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/122721

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
milestone: none → juno-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/122721
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2418a9dce86280fbf269f68f7d7c9b739f662ef1
Submitter: Jenkins
Branch: master

commit 2418a9dce86280fbf269f68f7d7c9b739f662ef1
Author: Daniel P. Berrange <email address hidden>
Date: Fri Sep 19 13:41:54 2014 +0100

    libvirt: avoid changing UUID when redefining nwfilters

    libvirt >= 1.2.7 enforces that when you re-define a network
    filter you can't change the UUID. ie name + uuid must match.
    Since Nova was not including any UUID in the XML it sent, it
    would always get a random UUID generated, which would cause
    failures when re-defining an existing filter. The result
    was that Nova would fail to start up and fail to migrate
    if there was an existing guest running. The fix is to query
    libvirt to see if the nwfilter already exists, and extract
    the UUID from its XML and use that when re-defining it.

    Closes-bug: #1360119
    Change-Id: I9d4b2c6c8f0c9a23ed79ed8e0b5ac0d4418851a4

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-rc1 → 2014.2
tags: added: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/167144

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/167144
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=68ec6840dd75dea85b6a55310c487284d7bd579e
Submitter: Jenkins
Branch: stable/icehouse

commit 68ec6840dd75dea85b6a55310c487284d7bd579e
Author: Daniel P. Berrange <email address hidden>
Date: Fri Sep 19 13:41:54 2014 +0100

    libvirt: avoid changing UUID when redefining nwfilters

    libvirt >= 1.2.7 enforces that when you re-define a network
    filter you can't change the UUID. ie name + uuid must match.
    Since Nova was not including any UUID in the XML it sent, it
    would always get a random UUID generated, which would cause
    failures when re-defining an existing filter. The result
    was that Nova would fail to start up and fail to migrate
    if there was an existing guest running. The fix is to query
    libvirt to see if the nwfilter already exists, and extract
    the UUID from its XML and use that when re-defining it.

    (cherry picked from commit 2418a9dce86280fbf269f68f7d7c9b739f662ef1)

    Closes-bug: #1360119
    Change-Id: I9d4b2c6c8f0c9a23ed79ed8e0b5ac0d4418851a4

tags: added: in-stable-icehouse
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.