| 2016-07-12 02:06:39 |
shiliang |
bug |
|
|
added bug |
| 2016-07-12 02:06:48 |
shiliang |
fuel-plugin-contrail: assignee |
|
shiliang (shiliang) |
|
| 2016-07-12 02:33:18 |
shiliang |
affects |
fuel-plugin-contrail |
nova |
|
| 2016-07-12 02:34:44 |
shiliang |
nova: status |
New |
In Progress |
|
| 2016-10-06 16:45:30 |
Matt Riedemann |
nova: importance |
Undecided |
Medium |
|
| 2016-10-06 16:45:37 |
Matt Riedemann |
nominated for series |
|
nova/mitaka |
|
| 2016-10-06 16:45:37 |
Matt Riedemann |
bug task added |
|
nova/mitaka |
|
| 2016-10-06 16:45:37 |
Matt Riedemann |
nominated for series |
|
nova/newton |
|
| 2016-10-06 16:45:37 |
Matt Riedemann |
bug task added |
|
nova/newton |
|
| 2016-10-06 16:45:53 |
Matt Riedemann |
tags |
|
libvirt |
|
| 2016-10-06 16:46:42 |
Matt Riedemann |
summary |
Error updating resources for some node |
(libvirt) KeyError updating resources for some node, guest.uuid is not in BDM list |
|
| 2016-10-06 18:16:08 |
OpenStack Infra |
nova: assignee |
shiliang (shiliang) |
Dan Smith (danms) |
|
| 2016-10-18 04:44:54 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2016-10-18 08:38:14 |
OpenStack Infra |
nova/newton: status |
New |
In Progress |
|
| 2016-10-18 08:38:14 |
OpenStack Infra |
nova/newton: assignee |
|
Lee Yarwood (lyarwood) |
|
| 2016-10-20 16:05:33 |
OpenStack Infra |
nova/newton: status |
In Progress |
Fix Committed |
|
| 2016-12-01 15:49:37 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2016-12-01 15:49:46 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/mitaka |
|
| 2016-12-01 15:50:05 |
Edward Hope-Morley |
tags |
libvirt |
libvirt sts |
|
| 2016-12-01 15:57:05 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/newton |
|
| 2016-12-01 16:01:35 |
Edward Hope-Morley |
nova/mitaka: assignee |
|
Edward Hope-Morley (hopem) |
|
| 2016-12-01 18:30:04 |
OpenStack Infra |
nova/mitaka: status |
New |
In Progress |
|
| 2016-12-08 16:57:13 |
Corey Bryant |
bug task added |
|
cloud-archive/mitaka |
|
| 2016-12-08 16:57:15 |
Corey Bryant |
bug task added |
|
cloud-archive/newton |
|
| 2016-12-08 16:57:58 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2016-12-08 17:00:57 |
Corey Bryant |
cloud-archive/mitaka: importance |
Undecided |
High |
|
| 2016-12-08 17:00:57 |
Corey Bryant |
cloud-archive/mitaka: status |
New |
Triaged |
|
| 2016-12-08 17:01:40 |
Corey Bryant |
cloud-archive/mitaka: importance |
High |
Medium |
|
| 2016-12-08 17:01:48 |
Corey Bryant |
cloud-archive/newton: importance |
Undecided |
Medium |
|
| 2016-12-08 17:01:51 |
Corey Bryant |
cloud-archive/newton: status |
New |
Triaged |
|
| 2016-12-09 15:12:00 |
Lee Yarwood |
nova/mitaka: status |
In Progress |
Won't Fix |
|
| 2016-12-09 15:12:14 |
Lee Yarwood |
nova/newton: importance |
Undecided |
Medium |
|
| 2017-02-06 17:17:34 |
Alvaro Uria |
tags |
libvirt sts |
canonical-bootstack libvirt sts |
|
| 2017-02-06 17:17:43 |
Alvaro Uria |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2017-03-16 15:51:09 |
Edward Hope-Morley |
cloud-archive/newton: status |
Triaged |
Fix Released |
|
| 2017-03-16 15:51:17 |
Edward Hope-Morley |
cloud-archive/mitaka: assignee |
|
Edward Hope-Morley (hopem) |
|
| 2017-03-16 15:55:56 |
Edward Hope-Morley |
bug task added |
|
ubuntu |
|
| 2017-03-16 15:56:06 |
Louis Bouchard |
bug task added |
|
nova (Ubuntu) |
|
| 2017-03-16 15:56:18 |
Louis Bouchard |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-03-16 15:56:18 |
Louis Bouchard |
bug task added |
|
nova (Ubuntu Xenial) |
|
| 2017-03-16 15:56:18 |
Louis Bouchard |
bug task added |
|
Ubuntu Xenial |
|
| 2017-03-16 15:57:08 |
Edward Hope-Morley |
bug task deleted |
ubuntu |
|
|
| 2017-03-16 15:57:32 |
Edward Hope-Morley |
bug task deleted |
Ubuntu Xenial |
|
|
| 2017-03-16 16:03:43 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts |
canonical-bootstack libvirt sts sts-sru |
|
| 2017-03-23 11:48:35 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts sts-sru |
canonical-bootstack libvirt sts sts-sru-needed |
|
| 2017-03-31 09:30:56 |
Edward Hope-Morley |
nova (Ubuntu Xenial): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2017-03-31 09:57:45 |
Launchpad Janitor |
nova (Ubuntu): status |
New |
Confirmed |
|
| 2017-03-31 09:57:45 |
Launchpad Janitor |
nova (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2017-03-31 14:14:39 |
Edward Hope-Morley |
summary |
(libvirt) KeyError updating resources for some node, guest.uuid is not in BDM list |
[SRU] (libvirt) KeyError updating resources for some node, guest.uuid is not in BDM list |
|
| 2017-03-31 14:15:06 |
Edward Hope-Morley |
description |
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager [req-d5d5d486-b488-4429-bbb5-24c9f19ff2c0 - - - - -] Error updating resources for node controller.
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager Traceback (most recent call last):
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6726, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager rt.update_available_resource(context)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 500, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5728, in get_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7397, in _get_disk_over_committed_size_total
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager local_instances[guest.uuid], bdms[guest.uuid])
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager KeyError: '0a5c5743-9555-4dfd-b26e-198449ebeee5'
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager |
[Impact]
There currently exists a race condition whereby the compute resource_tracker periodic task polls extant instances and checks their BDMs which can occur prior to any mappings having yet been created e.g. root disk mapping for new instances. This patch ensures that instances without any BDMs are skipped.
[Test Case]
* deploy Openstack Mitaka with debug logging enabled (not essential but helps)
* create an instance
* delete its BDMs - pastebin.ubuntu.com/24287419/
* watch /var/log/nova/nova-compute.log on hypervisor hosting instance and wait for next resource_tracker tick
* ensure that exception mentioned in LP does not occur (happens after "Auditing locally available compute resources for node")
[Regression Potential]
The resource tracker information is used by the scheduler when deciding which compute hosts are able to have an instances scheduled to them. In this case the resource tracker would be skipping instances that would contribute to disk overcommit ratios. As such it is possible that that scheduler will have momentarily skewed information about resource consumption on that compute host until the next resource_tracker tick. Since the likelihood of this race condition occurring is hopefully slim and provided that users have a reasonable frequency for the resource_tracker, the likelihood of this becoming a long term problem is low since the issue will always be corrected by a subsequent tick (although if the compute host in question were saturated that would not be fixed until an instances was deleted or migrated).
--------
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager [req-d5d5d486-b488-4429-bbb5-24c9f19ff2c0 - - - - -] Error updating resources for node controller.
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager Traceback (most recent call last):
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6726, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager rt.update_available_resource(context)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 500, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5728, in get_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7397, in _get_disk_over_committed_size_total
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager local_instances[guest.uuid], bdms[guest.uuid])
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager KeyError: '0a5c5743-9555-4dfd-b26e-198449ebeee5'
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager |
|
| 2017-03-31 14:16:02 |
Edward Hope-Morley |
attachment added |
|
lp1602057-xenial-mitaka.debdiff https://bugs.launchpad.net/nova/+bug/1602057/+attachment/4851346/+files/lp1602057-xenial-mitaka.debdiff |
|
| 2017-03-31 14:16:39 |
Edward Hope-Morley |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2017-03-31 14:17:05 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts sts-sru-needed |
canonical-bootstack libvirt sts sts-sponsor sts-sru-needed |
|
| 2017-03-31 14:22:38 |
Edward Hope-Morley |
description |
[Impact]
There currently exists a race condition whereby the compute resource_tracker periodic task polls extant instances and checks their BDMs which can occur prior to any mappings having yet been created e.g. root disk mapping for new instances. This patch ensures that instances without any BDMs are skipped.
[Test Case]
* deploy Openstack Mitaka with debug logging enabled (not essential but helps)
* create an instance
* delete its BDMs - pastebin.ubuntu.com/24287419/
* watch /var/log/nova/nova-compute.log on hypervisor hosting instance and wait for next resource_tracker tick
* ensure that exception mentioned in LP does not occur (happens after "Auditing locally available compute resources for node")
[Regression Potential]
The resource tracker information is used by the scheduler when deciding which compute hosts are able to have an instances scheduled to them. In this case the resource tracker would be skipping instances that would contribute to disk overcommit ratios. As such it is possible that that scheduler will have momentarily skewed information about resource consumption on that compute host until the next resource_tracker tick. Since the likelihood of this race condition occurring is hopefully slim and provided that users have a reasonable frequency for the resource_tracker, the likelihood of this becoming a long term problem is low since the issue will always be corrected by a subsequent tick (although if the compute host in question were saturated that would not be fixed until an instances was deleted or migrated).
--------
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager [req-d5d5d486-b488-4429-bbb5-24c9f19ff2c0 - - - - -] Error updating resources for node controller.
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager Traceback (most recent call last):
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6726, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager rt.update_available_resource(context)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 500, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5728, in get_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7397, in _get_disk_over_committed_size_total
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager local_instances[guest.uuid], bdms[guest.uuid])
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager KeyError: '0a5c5743-9555-4dfd-b26e-198449ebeee5'
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager |
[Impact]
There currently exists a race condition whereby the compute resource_tracker periodic task polls extant instances and checks their BDMs which can occur prior to any mappings having yet been created e.g. root disk mapping for new instances. This patch ensures that instances without any BDMs are skipped.
[Test Case]
* deploy Openstack Mitaka with debug logging enabled (not essential but helps)
* create an instance
* delete its BDMs - pastebin.ubuntu.com/24287419/
* watch /var/log/nova/nova-compute.log on hypervisor hosting instance and wait for next resource_tracker tick
* ensure that exception mentioned in LP does not occur (happens after "Auditing locally available compute resources for node")
[Regression Potential]
The resource tracker information is used by the scheduler when deciding which compute hosts are able to have an instances scheduled to them. In this case the resource tracker would be skipping instances that would contribute to disk overcommit ratios. As such it is possible that that scheduler will have momentarily skewed information about resource consumption on that compute host until the next resource_tracker tick. Since the likelihood of this race condition occurring is hopefully slim and provided that users have a reasonable frequency for the resource_tracker, the likelihood of this becoming a long term problem is low since the issue will always be corrected by a subsequent tick (although if the compute host in question were saturated that would not be fixed until an instances was deleted or migrated).
[Other]
Note that this patch did not make it into upstream stable/mitaka branch due to the stable cutoff so the proposal is to carry in the archive (indefinitely).
--------
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager [req-d5d5d486-b488-4429-bbb5-24c9f19ff2c0 - - - - -] Error updating resources for node controller.
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager Traceback (most recent call last):
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6726, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager rt.update_available_resource(context)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 500, in update_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager resources = self.driver.get_available_resource(self.nodename)
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5728, in get_available_resource
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7397, in _get_disk_over_committed_size_total
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager local_instances[guest.uuid], bdms[guest.uuid])
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager KeyError: '0a5c5743-9555-4dfd-b26e-198449ebeee5'
2016-07-12 09:54:36.021 10056 ERROR nova.compute.manager |
|
| 2017-03-31 15:34:30 |
Edward Hope-Morley |
attachment added |
|
lp1602057-xenial-mitaka.debdiff https://bugs.launchpad.net/nova/+bug/1602057/+attachment/4851390/+files/lp1602057-xenial-mitaka.debdiff |
|
| 2017-03-31 15:35:03 |
Edward Hope-Morley |
attachment removed |
lp1602057-xenial-mitaka.debdiff https://bugs.launchpad.net/nova/+bug/1602057/+attachment/4851346/+files/lp1602057-xenial-mitaka.debdiff |
|
|
| 2017-03-31 15:50:33 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts sts-sponsor sts-sru-needed |
canonical-bootstack libvirt sts sts-sru-needed |
|
| 2017-04-02 06:56:12 |
Mathew Hodson |
nova (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-04-02 06:56:18 |
Mathew Hodson |
nova (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2017-04-05 22:59:33 |
Brian Murray |
nova (Ubuntu Xenial): status |
Confirmed |
Incomplete |
|
| 2017-04-05 22:59:38 |
Brian Murray |
bug |
|
|
added subscriber Brian Murray |
| 2017-04-19 10:08:32 |
James Page |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2017-04-19 10:09:11 |
James Page |
nova (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2017-04-19 10:09:20 |
James Page |
nova (Ubuntu Xenial): status |
Incomplete |
Triaged |
|
| 2017-04-19 10:09:30 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-05-04 20:48:28 |
Brian Murray |
nova (Ubuntu Xenial): status |
Triaged |
Fix Committed |
|
| 2017-05-04 20:48:36 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2017-05-04 20:48:42 |
Brian Murray |
tags |
canonical-bootstack libvirt sts sts-sru-needed |
canonical-bootstack libvirt sts sts-sru-needed verification-needed |
|
| 2017-05-10 13:35:36 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts sts-sru-needed verification-needed |
canonical-bootstack libvirt sts sts-sru-needed verification-done |
|
| 2017-05-15 08:25:51 |
James Page |
cloud-archive/mitaka: status |
Triaged |
Fix Committed |
|
| 2017-05-15 08:25:54 |
James Page |
tags |
canonical-bootstack libvirt sts sts-sru-needed verification-done |
canonical-bootstack libvirt sts sts-sru-needed verification-done verification-mitaka-needed |
|
| 2017-05-17 01:40:46 |
Launchpad Janitor |
nova (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-05-17 01:40:56 |
Chris Halse Rogers |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-05-17 08:12:01 |
James Page |
cloud-archive/mitaka: status |
Fix Committed |
Fix Released |
|
| 2017-05-18 11:44:02 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts sts-sru-needed verification-done verification-mitaka-needed |
canonical-bootstack libvirt sts sts-sru-needed verification-done verification-mitaka-done |
|
| 2017-06-12 08:50:44 |
Edward Hope-Morley |
tags |
canonical-bootstack libvirt sts sts-sru-needed verification-done verification-mitaka-done |
canonical-bootstack libvirt sts sts-sru-done verification-done verification-mitaka-done |
|
| 2018-11-20 09:47:22 |
Dick |
bug |
|
|
added subscriber Dick |
