| 2018-12-21 13:12:21 |
Corey Bryant |
bug |
|
|
added bug |
| 2018-12-21 13:12:59 |
Corey Bryant |
bug task added |
|
nova (Ubuntu) |
|
| 2018-12-21 13:13:12 |
Corey Bryant |
nova (Ubuntu): status |
New |
Triaged |
|
| 2018-12-21 13:13:19 |
Corey Bryant |
nova (Ubuntu): importance |
Undecided |
Critical |
|
| 2018-12-21 13:14:08 |
Corey Bryant |
nova (Ubuntu): assignee |
|
Corey Bryant (corey.bryant) |
|
| 2018-12-21 13:42:20 |
Corey Bryant |
nominated for series |
|
Ubuntu Bionic |
|
| 2018-12-21 13:42:20 |
Corey Bryant |
bug task added |
|
nova (Ubuntu Bionic) |
|
| 2018-12-21 13:42:20 |
Corey Bryant |
nominated for series |
|
Ubuntu Disco |
|
| 2018-12-21 13:42:20 |
Corey Bryant |
bug task added |
|
nova (Ubuntu Disco) |
|
| 2018-12-21 13:42:20 |
Corey Bryant |
nominated for series |
|
Ubuntu Cosmic |
|
| 2018-12-21 13:42:20 |
Corey Bryant |
bug task added |
|
nova (Ubuntu Cosmic) |
|
| 2018-12-21 13:44:43 |
Corey Bryant |
nova (Ubuntu Disco): importance |
Critical |
High |
|
| 2018-12-21 13:44:46 |
Corey Bryant |
nova (Ubuntu Cosmic): importance |
Undecided |
High |
|
| 2018-12-21 13:44:48 |
Corey Bryant |
nova (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2018-12-21 13:44:50 |
Corey Bryant |
nova (Ubuntu Bionic): status |
New |
Triaged |
|
| 2018-12-21 13:44:51 |
Corey Bryant |
nova (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2018-12-21 13:45:14 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2018-12-21 13:46:22 |
Corey Bryant |
summary |
nova rbd auth fallback attempts to use cinder auth_username with libvirt secret_uuid |
nova rbd auth fallback attempts to use cinder user with libvirt secret |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
nominated for series |
|
cloud-archive/ocata |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
bug task added |
|
cloud-archive/ocata |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
nominated for series |
|
cloud-archive/queens |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
bug task added |
|
cloud-archive/queens |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
nominated for series |
|
cloud-archive/rocky |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
bug task added |
|
cloud-archive/rocky |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
nominated for series |
|
cloud-archive/pike |
|
| 2018-12-21 13:47:15 |
Corey Bryant |
bug task added |
|
cloud-archive/pike |
|
| 2018-12-21 13:47:36 |
Corey Bryant |
nominated for series |
|
cloud-archive/stein |
|
| 2018-12-21 13:47:36 |
Corey Bryant |
bug task added |
|
cloud-archive/stein |
|
| 2018-12-21 13:48:00 |
Corey Bryant |
cloud-archive/ocata: importance |
Undecided |
High |
|
| 2018-12-21 13:48:00 |
Corey Bryant |
cloud-archive/ocata: status |
New |
Triaged |
|
| 2018-12-21 13:48:28 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2018-12-21 13:48:28 |
OpenStack Infra |
nova: assignee |
|
Corey Bryant (corey.bryant) |
|
| 2018-12-21 13:51:50 |
Corey Bryant |
cloud-archive/pike: importance |
Undecided |
High |
|
| 2018-12-21 13:51:50 |
Corey Bryant |
cloud-archive/pike: status |
New |
Triaged |
|
| 2018-12-21 13:52:09 |
Corey Bryant |
cloud-archive/queens: importance |
Undecided |
High |
|
| 2018-12-21 13:52:09 |
Corey Bryant |
cloud-archive/queens: status |
New |
Triaged |
|
| 2018-12-21 13:52:24 |
Corey Bryant |
cloud-archive/rocky: importance |
Undecided |
High |
|
| 2018-12-21 13:52:24 |
Corey Bryant |
cloud-archive/rocky: status |
New |
Triaged |
|
| 2018-12-21 13:52:46 |
Corey Bryant |
cloud-archive/stein: importance |
Undecided |
High |
|
| 2018-12-21 13:52:46 |
Corey Bryant |
cloud-archive/stein: status |
New |
Triaged |
|
| 2018-12-21 13:53:24 |
Corey Bryant |
summary |
nova rbd auth fallback attempts to use cinder user with libvirt secret |
[SRU] nova rbd auth fallback attempts to use cinder user with libvirt secret |
|
| 2018-12-21 13:53:40 |
Corey Bryant |
summary |
[SRU] nova rbd auth fallback attempts to use cinder user with libvirt secret |
[SRU] nova rbd auth fallback uses cinder user with libvirt secret |
|
| 2018-12-21 13:57:49 |
Corey Bryant |
description |
From David Ames (thedac), originally posted to https://bugs.launchpad.net/charm-nova-compute/+bug/1671422/comments/25:
Updating this bug. We may decide to move this elsewhere it at some point.
We have a deployment that was upgraded through to pike at which point it was noticed that nova instances with ceph backed volumes would not start.
The cinder key was manually added to the nova-compute nodes in /etc/ceph and with:
sudo virsh secret-define --file /tmp/cinder.secret
However, this did not resolve the problem. It appeared libvirt was trying to use a mixed pair of usernames and keys. It was using the cinder username but the nova-compute key.
Looking at nova's code it falls back to nova.conf when it does not have a secret_uuid from cinder but it was not setting the username correctly.
https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/volume/net.py#L74
The following seems to mitigate this as a temporary fix on nova-compute until we can come up with a complete plan:
https://pastebin.ubuntu.com/p/tGm7C7fpXT/
diff --git a/nova/virt/libvirt/volume/net.py b/nova/virt/libvirt/volume/net.py
index cec43ce93b..8b0148df0b 100644
--- a/nova/virt/libvirt/volume/net.py
+++ b/nova/virt/libvirt/volume/net.py
@@ -71,6 +71,7 @@ class LibvirtNetVolumeDriver(libvirt_volume.LibvirtBaseVolumeDriver):
else:
LOG.debug('Falling back to Nova configuration for RBD auth '
'secret_uuid value.')
+ conf.auth_username = CONF.libvirt.rbd_user
conf.auth_secret_uuid = CONF.libvirt.rbd_secret_uuid
# secret_type is always hard-coded to 'ceph' in cinder
conf.auth_secret_type = netdisk_properties['secret_type']
Apply to /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/net.py
We still need a migration plan to get from the topology with nova-compute directly related to ceph to the topology with cinder-ceph related to nova-compute using ceph-access which would populate cinder's secret_uuid.
It is possible we will need to carry the patch for existing instances. It may be worth getting that upstream as master has the same problem. |
[Impact]
From David Ames (thedac), originally posted to https://bugs.launchpad.net/charm-nova-compute/+bug/1671422/comments/25:
Updating this bug. We may decide to move this elsewhere it at some point.
We have a deployment that was upgraded through to pike at which point it was noticed that nova instances with ceph backed volumes would not start.
The cinder key was manually added to the nova-compute nodes in /etc/ceph and with:
sudo virsh secret-define --file /tmp/cinder.secret
However, this did not resolve the problem. It appeared libvirt was trying to use a mixed pair of usernames and keys. It was using the cinder username but the nova-compute key.
Looking at nova's code it falls back to nova.conf when it does not have a secret_uuid from cinder but it was not setting the username correctly.
https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/volume/net.py#L74
The following seems to mitigate this as a temporary fix on nova-compute until we can come up with a complete plan:
https://pastebin.ubuntu.com/p/tGm7C7fpXT/
diff --git a/nova/virt/libvirt/volume/net.py b/nova/virt/libvirt/volume/net.py
index cec43ce93b..8b0148df0b 100644
--- a/nova/virt/libvirt/volume/net.py
+++ b/nova/virt/libvirt/volume/net.py
@@ -71,6 +71,7 @@ class LibvirtNetVolumeDriver(libvirt_volume.LibvirtBaseVolumeDriver):
else:
LOG.debug('Falling back to Nova configuration for RBD auth '
'secret_uuid value.')
+ conf.auth_username = CONF.libvirt.rbd_user
conf.auth_secret_uuid = CONF.libvirt.rbd_secret_uuid
# secret_type is always hard-coded to 'ceph' in cinder
conf.auth_secret_type = netdisk_properties['secret_type']
Apply to /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/net.py
We still need a migration plan to get from the topology with nova-compute directly related to ceph to the topology with cinder-ceph related to nova-compute using ceph-access which would populate cinder's secret_uuid.
It is possible we will need to carry the patch for existing instances. It may be worth getting that upstream as master has the same problem.
[Test Case]
Upgrade a juju-deployed cloud with ceph backend for nova and cinder from pre-ocata to ocata or above. Ensure that nova instances with ceph backed volumes successfully start.
[Regression Potential]
The fix is minimal and will not be fixed in Ubuntu until it has been approved upstream. |
|
| 2018-12-21 13:58:40 |
Corey Bryant |
bug |
|
|
added subscriber David Ames |
| 2018-12-21 13:58:52 |
Corey Bryant |
bug |
|
|
added subscriber Canonical Field Critical |
| 2018-12-21 19:52:03 |
Matt Riedemann |
tags |
|
ceph libvirt volumes |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
nominated for series |
|
nova/queens |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
bug task added |
|
nova/queens |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
nominated for series |
|
nova/rocky |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
bug task added |
|
nova/rocky |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
nominated for series |
|
nova/ocata |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
bug task added |
|
nova/ocata |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
nominated for series |
|
nova/pike |
|
| 2018-12-21 19:52:14 |
Matt Riedemann |
bug task added |
|
nova/pike |
|
| 2018-12-21 19:52:30 |
Matt Riedemann |
nova/ocata: status |
New |
Triaged |
|
| 2018-12-21 19:52:40 |
Matt Riedemann |
nova/pike: status |
New |
Triaged |
|
| 2018-12-21 19:52:51 |
Matt Riedemann |
nova/queens: importance |
Undecided |
Medium |
|
| 2018-12-21 19:52:51 |
Matt Riedemann |
nova/queens: status |
New |
Triaged |
|
| 2018-12-21 19:52:58 |
Matt Riedemann |
nova: importance |
Undecided |
Medium |
|
| 2018-12-21 19:53:06 |
Matt Riedemann |
nova/ocata: importance |
Undecided |
Medium |
|
| 2018-12-21 19:53:16 |
Matt Riedemann |
nova/pike: importance |
Undecided |
Medium |
|
| 2018-12-21 19:53:27 |
Matt Riedemann |
nova/rocky: importance |
Undecided |
Medium |
|
| 2018-12-21 19:53:27 |
Matt Riedemann |
nova/rocky: status |
New |
Triaged |
|
| 2018-12-21 21:55:51 |
OpenStack Infra |
nova: assignee |
Corey Bryant (corey.bryant) |
Matt Riedemann (mriedem) |
|
| 2018-12-21 21:58:19 |
Matt Riedemann |
nova: assignee |
Matt Riedemann (mriedem) |
Corey Bryant (corey.bryant) |
|
| 2018-12-22 21:33:40 |
Xav Paice |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2018-12-22 21:33:50 |
Xav Paice |
tags |
ceph libvirt volumes |
canonical-bootstack ceph libvirt volumes |
|
| 2018-12-25 03:27:23 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2019-01-02 12:53:11 |
OpenStack Infra |
cloud-archive/rocky: status |
Triaged |
Fix Committed |
|
| 2019-01-02 14:07:29 |
Matt Riedemann |
nova/rocky: status |
Triaged |
Fix Released |
|
| 2019-01-02 14:07:29 |
Matt Riedemann |
nova/rocky: assignee |
|
Corey Bryant (corey.bryant) |
|
| 2019-01-02 14:07:51 |
Matt Riedemann |
nova/queens: status |
Triaged |
In Progress |
|
| 2019-01-02 14:07:51 |
Matt Riedemann |
nova/queens: assignee |
|
Corey Bryant (corey.bryant) |
|
| 2019-01-06 13:17:59 |
OpenStack Infra |
cloud-archive/queens: status |
Triaged |
Fix Committed |
|
| 2019-01-07 20:47:00 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-01-07 23:10:07 |
Launchpad Janitor |
nova (Ubuntu Disco): status |
Triaged |
Fix Released |
|
| 2019-01-08 19:18:26 |
Brian Murray |
nova (Ubuntu Cosmic): status |
Triaged |
Fix Committed |
|
| 2019-01-08 19:18:33 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2019-01-08 19:18:38 |
Brian Murray |
tags |
canonical-bootstack ceph libvirt volumes |
canonical-bootstack ceph libvirt verification-needed verification-needed-cosmic volumes |
|
| 2019-01-08 19:27:41 |
Brian Murray |
nova (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
| 2019-01-08 19:27:49 |
Brian Murray |
tags |
canonical-bootstack ceph libvirt verification-needed verification-needed-cosmic volumes |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic volumes |
|
| 2019-01-09 00:17:20 |
Corey Bryant |
cloud-archive: status |
Triaged |
Fix Committed |
|
| 2019-01-09 00:18:35 |
Corey Bryant |
tags |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic volumes |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed volumes |
|
| 2019-01-09 00:21:19 |
Corey Bryant |
cloud-archive/pike: status |
Triaged |
Fix Committed |
|
| 2019-01-09 00:21:24 |
Corey Bryant |
tags |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-rocky-needed volumes |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-rocky-needed volumes |
|
| 2019-01-09 00:21:26 |
Corey Bryant |
cloud-archive/ocata: status |
Triaged |
Fix Committed |
|
| 2019-01-09 04:28:45 |
Corey Bryant |
tags |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-rocky-needed volumes |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed volumes |
|
| 2019-02-12 00:00:34 |
David Ames |
tags |
canonical-bootstack ceph libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-pike-needed verification-queens-needed verification-rocky-needed volumes |
canonical-bootstack ceph libvirt verification-done-bionic verification-needed-cosmic verification-newton-done verification-ocata-done verification-pike-done verification-queens-done verification-rocky-done volumes |
|
| 2019-02-12 12:47:54 |
Corey Bryant |
cloud-archive/stein: status |
Fix Committed |
Fix Released |
|
| 2019-02-12 15:36:35 |
David Ames |
tags |
canonical-bootstack ceph libvirt verification-done-bionic verification-needed-cosmic verification-newton-done verification-ocata-done verification-pike-done verification-queens-done verification-rocky-done volumes |
canonical-bootstack ceph libvirt verification-done-bionic verification-done-cosmic verification-newton-done verification-ocata-done verification-pike-done verification-queens-done verification-rocky-done volumes |
|
| 2019-02-12 19:26:00 |
Launchpad Janitor |
nova (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-02-12 19:26:13 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-02-12 19:55:07 |
Corey Bryant |
cloud-archive/rocky: status |
Fix Committed |
Fix Released |
|
| 2019-02-18 10:14:03 |
Launchpad Janitor |
nova (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-02-19 13:48:09 |
Corey Bryant |
cloud-archive/queens: status |
Fix Committed |
Fix Released |
|
| 2019-02-19 13:49:18 |
Corey Bryant |
cloud-archive/pike: status |
Fix Committed |
Fix Released |
|
| 2019-02-19 13:50:21 |
Corey Bryant |
cloud-archive/ocata: status |
Fix Committed |
Fix Released |
|
| 2019-03-26 07:37:57 |
OpenStack Infra |
nova/ocata: status |
Triaged |
Fix Committed |
|
| 2019-04-26 10:03:51 |
Lee Yarwood |
nova/queens: status |
In Progress |
Fix Released |
|
| 2019-07-22 17:59:48 |
Corey Bryant |
nova/pike: status |
Triaged |
Fix Released |
|
| 2019-10-29 18:19:18 |
Corey Bryant |
nova/ocata: status |
Fix Committed |
Fix Released |
|
