[SRU] nova rbd auth fallback uses cinder user with libvirt secret
| Affects | Status | Importance | Assigned to | Milestone | ||
|---|---|---|---|---|---|---|
| OpenStack Compute (nova) |
Medium
|
Corey Bryant | ||||
| Ocata |
Medium
|
Unassigned | ||||
| Pike |
Medium
|
Unassigned | ||||
| Queens |
Medium
|
Corey Bryant | ||||
| Rocky |
Medium
|
Corey Bryant | ||||
| Ubuntu Cloud Archive | Status tracked in Stein | |||||
| Ocata |
High
|
Unassigned | ||||
| Pike |
High
|
Unassigned | ||||
| Queens |
High
|
Unassigned | ||||
| Rocky |
High
|
Unassigned | ||||
| Stein |
High
|
Unassigned | ||||
| nova (Ubuntu) | Status tracked in Disco | |||||
| Bionic |
High
|
Unassigned | ||||
| Cosmic |
High
|
Unassigned | ||||
| Disco |
High
|
Corey Bryant | ||||
Bug Description
[Impact]
From David Ames (thedac), originally posted to https:/
Updating this bug. We may decide to move this elsewhere it at some point.
We have a deployment that was upgraded through to pike at which point it was noticed that nova instances with ceph backed volumes would not start.
The cinder key was manually added to the nova-compute nodes in /etc/ceph and with:
sudo virsh secret-define --file /tmp/cinder.secret
However, this did not resolve the problem. It appeared libvirt was trying to use a mixed pair of usernames and keys. It was using the cinder username but the nova-compute key.
Looking at nova's code it falls back to nova.conf when it does not have a secret_uuid from cinder but it was not setting the username correctly.
https:/
The following seems to mitigate this as a temporary fix on nova-compute until we can come up with a complete plan:
https:/
diff --git a/nova/
index cec43ce93b.
--- a/nova/
+++ b/nova/
@@ -71,6 +71,7 @@ class LibvirtNetVolum
else:
+ conf.auth_username = CONF.libvirt.
# secret_type is always hard-coded to 'ceph' in cinder
Apply to /usr/lib/
We still need a migration plan to get from the topology with nova-compute directly related to ceph to the topology with cinder-ceph related to nova-compute using ceph-access which would populate cinder's secret_uuid.
It is possible we will need to carry the patch for existing instances. It may be worth getting that upstream as master has the same problem.
[Test Case]
Upgrade a juju-deployed cloud with ceph backend for nova and cinder from pre-ocata to ocata or above. Ensure that nova instances with ceph backed volumes successfully start.
[Regression Potential]
The fix is minimal and will not be fixed in Ubuntu until it has been approved upstream.
| Changed in nova (Ubuntu): | |
| status: | New → Triaged |
| importance: | Undecided → Critical |
| assignee: | nobody → Corey Bryant (corey.bryant) |
| Changed in nova (Ubuntu Disco): | |
| importance: | Critical → High |
| Changed in nova (Ubuntu Cosmic): | |
| importance: | Undecided → High |
| status: | New → Triaged |
| Changed in nova (Ubuntu Bionic): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| summary: |
- nova rbd auth fallback attempts to use cinder auth_username with libvirt - secret_uuid + nova rbd auth fallback attempts to use cinder user with libvirt secret |
| Changed in nova: | |
| assignee: | nobody → Corey Bryant (corey.bryant) |
| status: | New → In Progress |
| summary: |
- nova rbd auth fallback attempts to use cinder user with libvirt secret + [SRU] nova rbd auth fallback attempts to use cinder user with libvirt + secret |
| summary: |
- [SRU] nova rbd auth fallback attempts to use cinder user with libvirt - secret + [SRU] nova rbd auth fallback uses cinder user with libvirt secret |
| description: | updated |
| tags: | added: ceph libvirt volumes |
| Changed in nova: | |
| importance: | Undecided → Medium |
| Changed in nova: | |
| assignee: | Corey Bryant (corey.bryant) → Matt Riedemann (mriedem) |
| Changed in nova: | |
| assignee: | Matt Riedemann (mriedem) → Corey Bryant (corey.bryant) |
| tags: | added: canonical-bootstack |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 47b7c4f3cc582bf
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500
Ensure rbd auth fallback uses matching credentials
As of Ocata, cinder config is preferred for rbd auth values with a
fallback to nova values [1]. The fallback path, for the case when
rbd_user is configured in cinder.conf and rbd_secret_uuid is not
configured in cinder.conf, results in the mismatched use of cinder
rbd_user with nova rbd_secret_uuid.
This fixes that fallback path to use nova rbd_user from nova.conf
with rbd_secret_uuid from nova.conf.
[1] See commit f2d27f6a8afb628
Thanks to David Ames for this fix.
Change-Id: Ieba216275c07ab
Co-Authored-By: David Ames <email address hidden>
Closes-Bug: #1809454
| Changed in nova: | |
| status: | In Progress → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/rocky
commit f5d8ee1bfc3b7b9
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500
Ensure rbd auth fallback uses matching credentials
As of Ocata, cinder config is preferred for rbd auth values with a
fallback to nova values [1]. The fallback path, for the case when
rbd_user is configured in cinder.conf and rbd_secret_uuid is not
configured in cinder.conf, results in the mismatched use of cinder
rbd_user with nova rbd_secret_uuid.
This fixes that fallback path to use nova rbd_user from nova.conf
with rbd_secret_uuid from nova.conf.
[1] See commit f2d27f6a8afb628
Thanks to David Ames for this fix.
Change-Id: Ieba216275c07ab
Co-Authored-By: David Ames <email address hidden>
Closes-Bug: #1809454
(cherry picked from commit 47b7c4f3cc582bf
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit accef50f9648dc4
Author: Corey Bryant <email address hidden>
Date: Fri Dec 21 08:23:32 2018 -0500
Ensure rbd auth fallback uses matching credentials
As of Ocata, cinder config is preferred for rbd auth values with a
fallback to nova values [1]. The fallback path, for the case when
rbd_user is configured in cinder.conf and rbd_secret_uuid is not
configured in cinder.conf, results in the mismatched use of cinder
rbd_user with nova rbd_secret_uuid.
This fixes that fallback path to use nova rbd_user from nova.conf
with rbd_secret_uuid from nova.conf.
[1] See commit f2d27f6a8afb628
Thanks to David Ames for this fix.
Change-Id: Ieba216275c07ab
Co-Authored-By: David Ames <email address hidden>
Closes-Bug: #1809454
(cherry picked from commit 47b7c4f3cc582bf
(cherry picked from commit f5d8ee1bfc3b7b9
| Corey Bryant (corey.bryant) wrote : | #5 |
New packages have been uploaded to Ubuntu disco, cosmic and bionic with patches cherry-picked from upstream nova. cosmic and bionic are awaiting SRU review.
| Launchpad Janitor (janitor) wrote : | #6 |
This bug was fixed in the package nova - 2:19.0.
---------------
nova (2:19.0.
* d/p/ensure-
picked from upstream to ensure ceph backend continues to work for upgrades
from pre-Ocata (LP: #1809454).
-- Corey Bryant <email address hidden> Fri, 21 Dec 2018 09:20:12 -0500
| Changed in nova (Ubuntu Disco): | |
| status: | Triaged → Fix Released |
Hello Corey, or anyone else affected,
Accepted nova into cosmic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| Changed in nova (Ubuntu Cosmic): | |
| status: | Triaged → Fix Committed |
| tags: | added: verification-needed verification-needed-cosmic |
| Brian Murray (brian-murray) wrote : | #8 |
Hello Corey, or anyone else affected,
Accepted nova into bionic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
| Changed in nova (Ubuntu Bionic): | |
| status: | Triaged → Fix Committed |
| tags: | added: verification-needed-bionic |
| Corey Bryant (corey.bryant) wrote : | #9 |
Hello Corey, or anyone else affected,
Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.
Please help us by testing this new package. To enable the -proposed repository:
sudo add-apt-repository cloud-archive:
sudo apt-get update
Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-
Further information regarding the verification process can be found at https:/
| Changed in cloud-archive: | |
| status: | Triaged → Fix Committed |
| tags: | added: verification-rocky-needed |
| Corey Bryant (corey.bryant) wrote : | #10 |
Hello Corey, or anyone else affected,
Accepted nova into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.
Please help us by testing this new package. To enable the -proposed repository:
sudo add-apt-repository cloud-archive:
sudo apt-get update
Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-
Further information regarding the verification process can be found at https:/
| Corey Bryant (corey.bryant) wrote : | #11 |
Hello Corey, or anyone else affected,
Accepted nova into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.
Please help us by testing this new package. To enable the -proposed repository:
sudo add-apt-repository cloud-archive:
sudo apt-get update
Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-
Further information regarding the verification process can be found at https:/
| tags: | added: verification-pike-needed |
| Corey Bryant (corey.bryant) wrote : | #12 |
Hello Corey, or anyone else affected,
Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.
Please help us by testing this new package. To enable the -proposed repository:
sudo add-apt-repository cloud-archive:
sudo apt-get update
Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-
Further information regarding the verification process can be found at https:/
| tags: | added: verification-queens-needed |
Fix proposed to branch: master
Review: https:/