can not get nova version info on some public clouds, causing nova client hangs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
python-novaclient |
Fix Released
|
Critical
|
Daniel Wallace |
Bug Description
(Note: heavily edited now that fixes are posted to give complete picture)
novaclient v 2.27.0 introduces the idea of the CLI version negotiating with the server so that it will be able to show users only help and options for features that cloud they are running against actually supports based on microversion. This requires that the clouds support the version info GET call.
Much to the surprise of everyone in the development team, some times these API calls are blocked or hang. When testing against an upstream DevStack the following GETs all work (once you have a valid keystone token).
GET / - returns all versions
GET /v2 - returns a 300 redirecting to /v2/
GET /v2/ - returns the version info for /v2/
Many clouds have apparently deployed in such a way that GET /v2 just hangs. The reasons why are opaque. The novaclient code was written to try that (which was probably a bug). Adding a slash reduces roundtrips by 1, and works on more clouds. Exactly why a bunch of clouds hang here is unknown.
As of this writing Rackspace and RunAbove return a 401 when trying to fetch /v2/ or /. That's coming out of they layer in front of Nova. That's definitely wrong behavior. This has now been directly reported back up to both orgs. Hopefully that will be addressed in the near future.
The fixes applied are as follows:
* https:/
* https:/
We should and will ensure there are Tempest tests that express the expected behavior here, and will get those into defcore to ensure that we're communicating that this is something we believe clouds must expose to be OpenStack. This whole process has been an series of surprises that things were not working in the field.
Some initial data about the state of various clouds is here -
http://
(Note: This bug also exposed the fact that Nova developers don't have credentials to many if any public clouds. HP employees have access to their cloud, RAX employees to theirs (since RAX free cloud access for devs started expiring 6 months ago). There aren't many actively engaged developers from other public clouds. So getting information about how things like clients operate on deployed clouds depends on proxying through individuals like Monty who have personally opened up accounts on many OpenStack clouds with their own credit card to attempt to test compatibility.)
Also note, infra's used of HP and RAX for providers meant that it was discovered on those clouds very soon after the novaclient release (which is consumed by nodepool). And that this code working on both of those clouds was considered a critical bug to get infra working again. If other clouds contributed resources to infra, their cloud would also be considered a "must work in novaclient" cloud, and compat issues found and addressed early.
Changed in nova: | |
importance: | Undecided → Critical |
status: | New → Confirmed |
milestone: | none → liberty-rc1 |
Changed in nova: | |
importance: | Critical → High |
Changed in python-novaclient: | |
status: | New → Confirmed |
no longer affects: | nova |
Changed in python-novaclient: | |
importance: | Undecided → High |
Changed in python-novaclient: | |
importance: | High → Critical |
tags: | added: liberty-rc-blocker |
Changed in python-novaclient: | |
assignee: | Sean Dague (sdague) → Matt Riedemann (mriedem) |
Changed in python-novaclient: | |
assignee: | Matt Riedemann (mriedem) → Sean Dague (sdague) |
summary: |
- against all sanity, nova needs to work around broken public clouds + can not get nova version info on some public clouds, causing nova client + hangs |
description: | updated |
Changed in python-novaclient: | |
milestone: | none → 2.28.0 |
status: | Fix Committed → Fix Released |
Changed in python-novaclient: | |
assignee: | Sean Dague (sdague) → Daniel Wallace (danielwallace) |
status: | Confirmed → In Progress |
Changed in python-novaclient: | |
milestone: | none → 2.29.0 |
status: | Fix Committed → Fix Released |
The novaclient release version in question is 2.27.0.