get_all_bw_usage raises keyerror

Bug #918383 reported by Aaron Lee
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Mohammed Naser

Bug Description

get_all_bw_usage raises a keyerror from deep within a code knot. I used Quantum and Melange while generating this.

ERROR nova.manager [-] Error during ComputeManager._poll_bandwidth_usage: u'0'
Traceback (most recent call last):
  File "/home/aaron/openstack/nova/nova/manager.py", line 155, in periodic_tasks
    task(self, context)
  File "/home/aaron/openstack/nova/nova/compute/manager.py", line 2016, in _poll_bandwidth_usage
    bw_usage = self.driver.get_all_bw_usage(start_time, stop_time)
  File "/home/aaron/openstack/nova/nova/virt/xenapi_conn.py", line 321, in get_all_bw_usage
    for iusage in self._vmops.get_all_bw_usage(start_time, stop_time).\
  File "/home/aaron/openstack/nova/nova/virt/xenapi/vmops.py", line 1370, in get_all_bw_usage
    vif_bw = vifs_bw.setdefault(vif_map[vname], {})(nova.manager):
KeyError: u'0'

Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Changed in nova:
assignee: nobody → Philip Knouff (philip-knouff)
status: Confirmed → In Progress
Brian Lamar (blamar)
Changed in nova:
assignee: Philip Knouff (philip-knouff) → Brian Lamar (blamar)
Revision history for this message
Brian Lamar (blamar) wrote :

Aaron: Can you provide any more information as to how this error can be reproduced? I'm not seeing this in my test environment and I'm curious how I might be able to duplicate it. Also, it seems like this code has been updated recently, so the traceback isn't matching up with the current code, making it a bit more difficult to see if this is still an issue.

Brian Lamar (blamar)
Changed in nova:
status: In Progress → Incomplete
Revision history for this message
Aaron Lee (aaron-lee) wrote :

I believe Dragon fixed this issue while working on Yaggi..

Revision history for this message
Thierry Carrez (ttx) wrote :

Closing per previous comment, reopen if this is still current.

Changed in nova:
status: Incomplete → Invalid
Revision history for this message
Mohammed Naser (mnaser) wrote :

I'm seeing this problem as well, every poll where the usage cache has to be updated fails (trunk):

# grep -A 12 'Updating bandwidth usage cache' /var/log/nova/nova-compute.log
2012-07-16 06:53:18 INFO nova.compute.manager [-] Updating bandwidth usage cache
2012-07-16 06:53:18 ERROR nova.manager [-] Error during ComputeManager._poll_bandwidth_usage: u'1'
2012-07-16 06:53:18 TRACE nova.manager Traceback (most recent call last):
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/manager.py", line 167, in periodic_tasks
2012-07-16 06:53:18 TRACE nova.manager task(self, context)
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/compute/manager.py", line 2437, in _poll_bandwidth_usage
2012-07-16 06:53:18 TRACE nova.manager stop_time)
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/virt/xenapi/driver.py", line 308, in get_all_bw_usage
2012-07-16 06:53:18 TRACE nova.manager iusages = self._vmops.get_all_bw_usage(start_time, stop_time)
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/virt/xenapi/vmops.py", line 1195, in get_all_bw_usage
2012-07-16 06:53:18 TRACE nova.manager vif_bw = vifs_bw.setdefault(vif_map[vname], {})
2012-07-16 06:53:18 TRACE nova.manager KeyError: u'1'
2012-07-16 06:53:18 TRACE nova.manager

Changed in nova:
status: Invalid → Confirmed
Revision history for this message
Mohammed Naser (mnaser) wrote :

Okay, after some debugging. It seems that it tries to run for all of them virtual machines, including the one running nova-compute. I'll be refactoring it so that it only gets nova vms (the ones with nova_uuid in other_config) is a good source.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/9874

Changed in nova:
assignee: Brian Lamar (blamar) → Mohammed Naser (mnaser)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/9874
Committed: http://github.com/openstack/nova/commit/212045e4c87aee81fde82dd4887d321cb511d717
Submitter: Jenkins
Branch: master

commit 212045e4c87aee81fde82dd4887d321cb511d717
Author: Mohammed Naser <email address hidden>
Date: Mon Jul 16 17:57:29 2012 -0400

    Improved VM detection for bandwidth polling (XAPI)

    XenAPI bandwidth polling would fail or be unreliable if there
    was any other virtual machines on the same node (such as the nova
    compute node). This uses the nova_uuid which is much more
    reliable to be there. Fixes LP Bug #918383

    Change-Id: I1eb9e6289a8171d6fc93a49b327df474b1e2e503

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-3 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.