get_all_bw_usage raises keyerror

[Aaron Lee]

get_all_bw_usage raises a keyerror from deep within a code knot. I used Quantum and Melange while generating this.

ERROR nova.manager [-] Error during ComputeManager._poll_bandwidth_usage: u'0'
Traceback (most recent call last):
  File "/home/aaron/openstack/nova/nova/manager.py", line 155, in periodic_tasks
    task(self, context)
  File "/home/aaron/openstack/nova/nova/compute/manager.py", line 2016, in _poll_bandwidth_usage
    bw_usage = self.driver.get_all_bw_usage(start_time, stop_time)
  File "/home/aaron/openstack/nova/nova/virt/xenapi_conn.py", line 321, in get_all_bw_usage
    for iusage in self._vmops.get_all_bw_usage(start_time, stop_time).\
  File "/home/aaron/openstack/nova/nova/virt/xenapi/vmops.py", line 1370, in get_all_bw_usage
    vif_bw = vifs_bw.setdefault(vif_map[vname], {})(nova.manager):
KeyError: u'0'

[Brian Lamar (blamar)]

Aaron: Can you provide any more information as to how this error can be reproduced? I'm not seeing this in my test environment and I'm curious how I might be able to duplicate it. Also, it seems like this code has been updated recently, so the traceback isn't matching up with the current code, making it a bit more difficult to see if this is still an issue.

[Aaron Lee (aaron-lee)]

I believe Dragon fixed this issue while working on Yaggi..

[Thierry Carrez (ttx)]

Closing per previous comment, reopen if this is still current.

[Mohammed Naser (mnaser)]

I'm seeing this problem as well, every poll where the usage cache has to be updated fails (trunk):

# grep -A 12 'Updating bandwidth usage cache' /var/log/nova/nova-compute.log
2012-07-16 06:53:18 INFO nova.compute.manager [-] Updating bandwidth usage cache
2012-07-16 06:53:18 ERROR nova.manager [-] Error during ComputeManager._poll_bandwidth_usage: u'1'
2012-07-16 06:53:18 TRACE nova.manager Traceback (most recent call last):
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/manager.py", line 167, in periodic_tasks
2012-07-16 06:53:18 TRACE nova.manager task(self, context)
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/compute/manager.py", line 2437, in _poll_bandwidth_usage
2012-07-16 06:53:18 TRACE nova.manager stop_time)
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/virt/xenapi/driver.py", line 308, in get_all_bw_usage
2012-07-16 06:53:18 TRACE nova.manager iusages = self._vmops.get_all_bw_usage(start_time, stop_time)
2012-07-16 06:53:18 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/virt/xenapi/vmops.py", line 1195, in get_all_bw_usage
2012-07-16 06:53:18 TRACE nova.manager vif_bw = vifs_bw.setdefault(vif_map[vname], {})
2012-07-16 06:53:18 TRACE nova.manager KeyError: u'1'
2012-07-16 06:53:18 TRACE nova.manager

[Mohammed Naser (mnaser)]

Okay, after some debugging. It seems that it tries to run for all of them virtual machines, including the one running nova-compute. I'll be refactoring it so that it only gets nova vms (the ones with nova_uuid in other_config) is a good source.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/9874

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/9874
Committed: http://github.com/openstack/nova/commit/212045e4c87aee81fde82dd4887d321cb511d717
Submitter: Jenkins
Branch: master

commit 212045e4c87aee81fde82dd4887d321cb511d717
Author: Mohammed Naser <email address hidden>
Date: Mon Jul 16 17:57:29 2012 -0400

    Improved VM detection for bandwidth polling (XAPI)

    XenAPI bandwidth polling would fail or be unreliable if there
    was any other virtual machines on the same node (such as the nova
    compute node). This uses the nova_uuid which is much more
    reliable to be there. Fixes LP Bug #918383

    Change-Id: I1eb9e6289a8171d6fc93a49b327df474b1e2e503