horizon fails authentication with keystone

[Jeremy Nauck]

This report is almost identical to https://bugs.launchpad.net/horizon/+bug/1637072

Chef allinone installation following directions here; https://github.com/openstack/openstack-chef-repo

Command line access works fine, but access from Horizon to many functions renders an error similar to "Error: Unable to retrieve usage information." in System -> Overview. Similarly trying to display flavors yields the error "Error: Unable to retrieve flavor list."

Attempting to create a Flavor will cause an exception with the message "Danger: There was an error submitting the form. Please try again."

This installation is running Newton on Ubuntu 16.04 with Horizon 10.0.

The Horizon operations are frequently logging errors like these when trying to create a new Flavor.

DEBUG:keystoneauth.session:GET call to identity for http://127.0.0.1:5000/v3/users/e87dfdcd37204e60b9cc5e2b4f2cecee/projects used request id req-929b0e41-1ec0-4435-b8e9-de0ea821668d
DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://127.0.0.1:5000/v3/auth/token
DEBUG:keystoneauth.session:Request returned failure status: 400
Internal Server Error: /admin/flavors/create/
....
[Fri Feb 03 00:55:32.998833 2017] [wsgi:error] [pid 4156:tid 140316095899392] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 655, in request
[Fri Feb 03 00:55:32.998835 2017] [wsgi:error] [pid 4156:tid 140316095899392] raise exceptions.from_response(resp, method, url)
[Fri Feb 03 00:55:32.998837 2017] [wsgi:error] [pid 4156:tid 140316095899392] BadRequest: Expecting to find domain in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-8e28b2a0-25c4-466c-b881-47a0cd8224a1)

After following along in the Bug report 1637072 I attempted to replace python-novaclient with 7.1.0, but the error still persisted. I'm currently running python-novaclient 6.0.0 and it also seems to fail with similar errors.

[Jeremy Nauck (jedihamster)]

Ok, some progress... I still had some lingering 7.1.0 artifacts. Cleaned everything up and reinstalled v6.0.0 and things seem much better now.

The analysis in the last bug report was accurate. The Chef install is pulling a later version of the python-novaclient and it doesn't seem to be compatible with Newton & Horizon v10.

[Gianpietro Lavado (gianpietro.lavado)]

I hit this bug from a component which I would think is unrelated (tacker), so it was difficult to narrow the problem, glad to find this bug. Yesterday I installed a newer version of Tacker, which also pulled this new version of python-novaclient. Now I'm not able to perform any nova-related tasks from horizon (unable to retrieve instances, unable to retrieve the flavors, etc).

This is my log from apache:

[Tue Feb 14 08:56:53.710637 2017] [wsgi:error] [pid 2001:tid 139696932181760] Recoverable error: Expecting to find domain in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-08731c0b-05a9-4b28-9959-fbed98f2308d)
[Tue Feb 14 08:56:54.149827 2017] [wsgi:error] [pid 2001:tid 139696932181760] Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will not be displayed.
[Tue Feb 14 08:56:54.221511 2017] [wsgi:error] [pid 2001:tid 139696932181760] Failed to retrieve quota information
[Tue Feb 14 08:56:54.221572 2017] [wsgi:error] [pid 2001:tid 139696932181760] Traceback (most recent call last):
[Tue Feb 14 08:56:54.221591 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/tables.py", line 389, in allowed
[Tue Feb 14 08:56:54.221609 2017] [wsgi:error] [pid 2001:tid 139696932181760] limits = api.nova.tenant_absolute_limits(request, reserved=True)
[Tue Feb 14 08:56:54.221622 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 949, in tenant_absolute_limits
[Tue Feb 14 08:56:54.221633 2017] [wsgi:error] [pid 2001:tid 139696932181760] limits = novaclient(request).limits.get(reserved=reserved).absolute
[Tue Feb 14 08:56:54.221645 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
[Tue Feb 14 08:56:54.221657 2017] [wsgi:error] [pid 2001:tid 139696932181760] return self._get("/limits%s" % query_string, "limits")
[Tue Feb 14 08:56:54.221669 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 351, in _get
[Tue Feb 14 08:56:54.221680 2017] [wsgi:error] [pid 2001:tid 139696932181760] resp, body = self.api.client.get(url)
[Tue Feb 14 08:56:54.221691 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 217, in get
[Tue Feb 14 08:56:54.221703 2017] [wsgi:error] [pid 2001:tid 139696932181760] return self.request(url, 'GET', **kwargs)
[Tue Feb 14 08:56:54.221712 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 74, in request
[Tue Feb 14 08:56:54.221723 2017] [wsgi:error] [pid 2001:tid 139696932181760] **kwargs)
[Tue Feb 14 08:56:54.221734 2017] [wsgi:error] [pid 2001:tid 139696932181760] File "/usr/local/lib/pyt...

[Richard Jones (r1chardj0n3s)]

Please ensure that your installed libraries match those specified in the upper-constraints.txt file relevant to the version of Horizon you're running (see https://docs.openstack.org/developer/horizon/topics/install.html for how that works).

[Marcio Cavalcante (mcavalcante)]

I´m with this same problem, in both Tacker and Horizon. I tried clearing all the installating and reinstalled Tacker changing the file requirements.txt with novaclient beeing 6.0.0, but no success. It was raised another problem with packet cryptography during the compilation (gcc).

It worth mentioning that I found out that the online Openstack installation procedure has changed late last week. I reinstalled openstack from scratch twice last week (20/feb and 25/feb) , and the last one was with some changes, for example using fernet. In both of them I caught the same problem.

[Alejandro Comisario (alejandro-f)]

any news related on this ?