| 2015-09-02 20:37:49 |
Monty Taylor |
bug |
|
|
added bug |
| 2015-09-02 20:39:03 |
Sean Dague |
nova: importance |
Undecided |
Critical |
|
| 2015-09-02 20:39:06 |
Sean Dague |
nova: status |
New |
Confirmed |
|
| 2015-09-02 20:39:13 |
Sean Dague |
nova: milestone |
|
liberty-rc1 |
|
| 2015-09-02 20:39:28 |
Matt Riedemann |
nova: importance |
Critical |
High |
|
| 2015-09-02 20:39:37 |
Matt Riedemann |
bug task added |
|
python-novaclient |
|
| 2015-09-02 20:39:46 |
Matt Riedemann |
python-novaclient: status |
New |
Confirmed |
|
| 2015-09-02 20:39:50 |
Matt Riedemann |
bug task deleted |
nova |
|
|
| 2015-09-02 20:39:55 |
Matt Riedemann |
python-novaclient: importance |
Undecided |
High |
|
| 2015-09-02 20:40:01 |
Sean Dague |
python-novaclient: importance |
High |
Critical |
|
| 2015-09-02 20:40:09 |
Matt Riedemann |
tags |
|
liberty-rc-blocker |
|
| 2015-09-02 21:49:04 |
Jesse Keating |
bug |
|
|
added subscriber Jesse Keating |
| 2015-09-02 22:02:54 |
Jim Meyer |
bug |
|
|
added subscriber Jim Meyer |
| 2015-09-03 12:08:23 |
Doug Hellmann |
bug |
|
|
added subscriber Doug Hellmann |
| 2015-09-03 12:28:36 |
OpenStack Infra |
python-novaclient: status |
Confirmed |
In Progress |
|
| 2015-09-03 12:28:36 |
OpenStack Infra |
python-novaclient: assignee |
|
Sean Dague (sdague) |
|
| 2015-09-03 14:33:41 |
OpenStack Infra |
python-novaclient: assignee |
Sean Dague (sdague) |
Matt Riedemann (mriedem) |
|
| 2015-09-03 17:02:56 |
Justin R. Knierim |
bug |
|
|
added subscriber Justin R. Knierim |
| 2015-09-03 17:10:54 |
David-wahlstrom |
bug |
|
|
added subscriber David-wahlstrom |
| 2015-09-03 17:19:38 |
OpenStack Infra |
python-novaclient: assignee |
Matt Riedemann (mriedem) |
Sean Dague (sdague) |
|
| 2015-09-03 17:44:11 |
Sean Dague |
summary |
against all sanity, nova needs to work around broken public clouds |
can not get nova version info on some public clouds, causing nova client hangs |
|
| 2015-09-03 18:03:26 |
Sean Dague |
description |
The version listing endpoint is a clearly a giant security risk, so the intrepid operators of Rackspace, HP, Dreamhost, Auro and RunAbove have all decided, in their infinite wisdom and deference to magical ponies that BLOCKING it is a great idea.
Luckily, a non-zero number of them have decided to block it by hanging connections rather than throwing a 401.
http://paste.openstack.org/show/442384/
In any case, the end result is that those clouds are unusable with the latest release of novaclient, which means it's probably on the nova team to figure out a workaround.
After this bug is fixed, a defcore test should be added and it should be very non-optional.
Also, everyone involved with blocking the VERSION LISTING endpoint should feel very ashamed of the stance against progress, joy and light.
On the other hand, please everyone join me in rejoicing in the excellent work by the teams at vexxhost and UnitedStack who are running excellent clouds that do not exhibit this pox. |
(Note: heavily edited now that fixes are posted to give complete picture)
novaclient v 2.27.0 introduces the idea of the CLI version negotiating with the server so that it will be able to show users only help and options for features that cloud they are running against actually supports based on microversion. This requires that the clouds support the version info GET call.
Much to the surprise of everyone in the development team, some times these API calls are blocked or hang. When testing against an upstream DevStack the following GETs all work (once you have a valid keystone token).
GET / - returns all versions
GET /v2 - returns a 300 redirecting to /v2/
GET /v2/ - returns the version info for /v2/
Many clouds have apparently deployed in such a way that GET /v2 just hangs. The reasons why are opaque. The novaclient code was written to try that (which was probably a bug). Adding a slash reduces roundtrips by 1, and works on more clouds. Exactly why a bunch of clouds hang here is unknown.
As of this writing Rackspace and RunAbove return a 401 when trying to fetch /v2/ or /. That's coming out of they layer in front of Nova. That's definitely wrong behavior. This has now been directly reported back up to both orgs. Hopefully that will be addressed in the near future.
The fixes applied are as follows:
* https://review.openstack.org/#/c/220111 - this adds the extra '/' that works on more clouds.
* https://review.openstack.org/#/c/220192 - this adds 401 catching
We should and will ensure there are Tempest tests that express the expected behavior here, and will get those into defcore to ensure that we're communicating that this is something we believe clouds must expose to be OpenStack. This whole process has been an series of surprises that things were not working in the field.
Some initial data about the state of various clouds is here -
http://paste.openstack.org/show/442384/ - though it turns out that Monty has misconfigured Dreamhost credentials, so the Dreamhost data there is not valid.
(Note: This bug also exposed the fact that Nova developers don't have credentials to many if any public clouds. HP employees have access to their cloud, RAX employees to theirs (since RAX free cloud access for devs started expiring 6 months ago). There aren't many actively engaged developers from other public clouds. So getting information about how things like clients operate on deployed clouds depends on proxying through individuals like Monty who have personally opened up accounts on many OpenStack clouds with their own credit card to attempt to test compatibility.)
Also note, infra's used of HP and RAX for providers meant that it was discovered on those clouds very soon after the novaclient release (which is consumed by nodepool). And that this code working on both of those clouds was considered a critical bug to get infra working again. If other clouds contributed resources to infra, their cloud would also be considered a "must work in novaclient" cloud, and compat issues found and addressed early. |
|
| 2015-09-03 22:08:51 |
Mathieu Gagné |
bug |
|
|
added subscriber Mathieu Gagné |
| 2015-09-03 22:39:00 |
OpenStack Infra |
python-novaclient: status |
In Progress |
Fix Committed |
|
| 2015-09-04 02:28:33 |
Davanum Srinivas (DIMS) |
python-novaclient: status |
Fix Committed |
Fix Released |
|
| 2015-09-04 02:28:33 |
Davanum Srinivas (DIMS) |
python-novaclient: milestone |
|
2.28.0 |
|
| 2015-09-09 19:14:52 |
melanie witt |
python-novaclient: status |
Fix Released |
Confirmed |
|
| 2015-09-09 19:14:52 |
melanie witt |
python-novaclient: milestone |
2.28.0 |
|
|
| 2015-09-09 19:29:58 |
OpenStack Infra |
python-novaclient: status |
Confirmed |
In Progress |
|
| 2015-09-09 19:29:58 |
OpenStack Infra |
python-novaclient: assignee |
Sean Dague (sdague) |
Daniel Wallace (danielwallace) |
|
| 2015-09-10 18:11:03 |
OpenStack Infra |
python-novaclient: status |
In Progress |
Fix Committed |
|
| 2015-09-11 14:46:57 |
Doug Hellmann |
python-novaclient: status |
Fix Committed |
Fix Released |
|
| 2015-09-11 14:46:57 |
Doug Hellmann |
python-novaclient: milestone |
|
2.29.0 |
|
