Horizon "Error: unable to retrieve volume list" when selecting Volumes item

[Eric Miller]

Kolla-Ansible version: 7.0.0.0b3
Kolla version: 7.0.0.0b3
Distro: CentOS 7.5 (latest updates) on both deployment node and all hosts
Kolla images built on July 29, 2018

As described in the Summary, an error is shown when clicking Volumes, but also when clicking Backups, Snapshots, and other Volume-related items.

The horizon.log file on the respective controller where the "horizon" container is located has the following error:
[Mon Jul 30 21:04:31.809698 2018] [:error] [pid 67] Recoverable error: API version '3.0' is not supported on 'cinderclient.v3.volume_transfers.list' method.

OpenStack CLI functions work fine, such as "volume list", "volume create", etc. where the backend is Ceph, configured by Kolla-Ansible, so this appears to be specifically a Horizon issue that did not exist in 7.0.0.0b2.

Eric

[Eduardo Gonzalez (egonzalez90)]

Hi, what image type are you using (source or binary)?
Also please share package list from inside horizon container and cinder-api.

docker exec horizon (pip freeze or rpm -qa)

My first guess is that cinder changed the API but changes are not reflected in the horizon part yet. We will need more logs, please set debug mode to horizon.

Regards

[Eduardo Gonzalez (egonzalez90)]

Changes may be related:

Cinder: https://review.openstack.org/#/c/533564/
Cinder client: https://review.openstack.org/#/c/577611/

[Eric Miller (erickmiller)]

Images were built with the "source" option.

Attached is a zip file called packages_list.zip with 4 files for both pip and yum installed packages for both containers (horizon and cinder_api). I thought these commands would be cleaner visually, but if you want pip freeze and/or rpm-qa, let me know and I'll re-run.

Note that this test environment has 3 controllers each with horizon and cinder_api containers, but the error happens on all of them.

I will set the debug mode to True in the globals.yml and re-deploy and forward the logs from horizon after reproducing the problem again.

[Eric Miller (erickmiller)]

I set debug to True, re-deployed on fresh VM nodes, and verified that the local_settings file had debug turned on - these are the settings:

DEBUG = True
TEMPLATE_DEBUG = DEBUG

but I got the same error information as before (no additional detail). Below is the entire log with 4 attempts to load volume-related items in Horizon shown at the bottom of the log.

Eric

[Tue Jul 31 15:06:00.606110 2018] [:error] [pid 68] WARNING:neutron_lbaas_dashboard.dashboards.project.ngloadbalancersv2.panel:neutron-lbaas-dashboard is now deprecated. See: https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation
[Tue Jul 31 15:06:26.736501 2018] [:error] [pid 70] WARNING:root:Use of this 'djano.wsgi' file has been deprecated since the Rocky release in favor of 'wsgi.py' in the 'openstack_dashboard' module. This file is a legacy naming from before Django 1.4 and an importable 'wsgi.py' is now the default. This file will be removed in the T release cycle.
[Tue Jul 31 15:06:29.411100 2018] [:error] [pid 70] WARNING:neutron_lbaas_dashboard.dashboards.project.ngloadbalancersv2.panel:neutron-lbaas-dashboard is now deprecated. See: https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation
[Tue Jul 31 15:06:37.480617 2018] [:error] [pid 70] Attempted scope to domain default failed, will attempt to scope to another domain.
[Tue Jul 31 15:06:44.745278 2018] [:error] [pid 70] Login successful for user "admin" using domain "default", remote address 192.168.4.3.
[Tue Jul 31 15:06:45.047088 2018] [:error] [pid 69] WARNING:root:Use of this 'djano.wsgi' file has been deprecated since the Rocky release in favor of 'wsgi.py' in the 'openstack_dashboard' module. This file is a legacy naming from before Django 1.4 and an importable 'wsgi.py' is now the default. This file will be removed in the T release cycle.
[Tue Jul 31 15:06:47.443278 2018] [:error] [pid 69] WARNING:neutron_lbaas_dashboard.dashboards.project.ngloadbalancersv2.panel:neutron-lbaas-dashboard is now deprecated. See: https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation
[Tue Jul 31 15:07:51.730165 2018] [:error] [pid 70] Attempted scope to domain default failed, will attempt to scope to another domain.
[Tue Jul 31 15:07:51.840242 2018] [:error] [pid 70] Login successful for user "admin" using domain "default", remote address 192.168.4.3.
[Tue Jul 31 15:08:34.484616 2018] [:error] [pid 68] Recoverable error: API version '3.0' is not supported on 'cinderclient.v3.volume_transfers.list' method.
[Tue Jul 31 15:10:20.625941 2018] [:error] [pid 70] Recoverable error: API version '3.0' is not supported on 'cinderclient.v3.volume_transfers.list' method.
[Tue Jul 31 15:10:25.148744 2018] [:error] [pid 68] Recoverable error: API version '3.0' is not supported on 'cinderclient.v3.volume_transfers.list' method.
[Tue Jul 31 15:10:28.057236 2018] [:error] [pid 67] Recoverable error: API version '3.0' is not supported on 'cinderclient.v3.volume_transfers.list' method.

[Eric Miller (erickmiller)]

In case this was of interest, this is the contents /var/log/kolla/cinder/cinder-api.log file in the cinder_api container. Looks like it just has some deprecation warnings, but maybe you can see something else that I missed?

2018-07-31 15:07:18.836254 /var/lib/kolla/venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
2018-07-31 15:07:18.836323 return pkg_resources.EntryPoint.parse("x=" + s).load(False)
2018-07-31 15:07:19.369570 /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
2018-07-31 15:07:19.369619 self._legacy_facade = LegacyEngineFacade(None, _factory=self)
2018-07-31 15:07:20.499420 /var/lib/kolla/venv/lib/python2.7/site-packages/webob/acceptparse.py:972: DeprecationWarning: The behavior of AcceptValidHeader.best_match is currently being maintained for backward compatibility, but it will be deprecated in the future, as it does not conform to the RFC.
2018-07-31 15:07:20.499537 DeprecationWarning,
2018-07-31 15:07:32.539871 /var/lib/kolla/venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
2018-07-31 15:07:32.539938 return pkg_resources.EntryPoint.parse("x=" + s).load(False)
2018-07-31 15:07:33.061870 /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
2018-07-31 15:07:33.061945 self._legacy_facade = LegacyEngineFacade(None, _factory=self)
2018-07-31 15:07:34.179508 /var/lib/kolla/venv/lib/python2.7/site-packages/webob/acceptparse.py:972: DeprecationWarning: The behavior of AcceptValidHeader.best_match is currently being maintained for backward compatibility, but it will be deprecated in the future, as it does not conform to the RFC.
2018-07-31 15:07:34.179589 DeprecationWarning,
2018-07-31 15:08:09.710418 /var/lib/kolla/venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
2018-07-31 15:08:09.710482 return pkg_resources.EntryPoint.parse("x=" + s).load(False)
2018-07-31 15:08:10.255893 /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:350: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
2018-07-31 15:08:10.255943 self._legacy_facade = LegacyEngineFacade(None, _factory=self)
2018-07-31 15:08:11.409800 /var/lib/kolla/venv/lib/python2.7/site-packages/webob/acceptparse.py:972: DeprecationWarning: The behavior of AcceptValidHeader.best_match is currently being maintained for backward compatibility, but it will be deprecated in the future, as it does not conform to the RFC.
2018-07-31 15:08:11.409858 DeprecationWarning,
2018-07-31 15:08:32.819689 /var/lib/kolla/venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are depreca...

[Akihiro Motoki (amotoki)]

When debug log is enabled for cinderclient in horizon, I got the following log with horizon Rocky 14.0.0.0rc1.

horizon cinderclient accessed Cinder v3 API and no microversion is specified.
In the response, Cinder API said API version 3.0 is used and volume transfer list was successfully retrieved.

DEBUG cinderclient.client
REQ: curl -i http://172.27.201.206/volume/v3/1e6f55d13b6147c199d5f4bb28bd09f2/os-volume-transfer/detail -X GET-H 'User-Agent: python-cinderclient'-H 'X-Auth-Project-Id: 1e6f55d13b6147c199d5f4bb28bd09f2'-H 'Accept: application/json'-H 'X-Auth-Token: {SHA1}a12541f786e94ae9c0553af1e3db4b49441fb9bf'

DEBUG cinderclient.client RESP: [200] {'x-openstack-request-id': 'req-d813a2f5-fbfe-4eab-9eaf-5bb2ee36a635', 'Date': 'Fri, 10 Aug 2018 21:04:34 GMT', 'Vary': 'OpenStack-API-Version', 'Content-Length': '17', 'Server': 'Apache/2.4.18 (Ubuntu)', 'OpenStack-API-Version': 'volume 3.0', 'Content-Type': 'application/json', 'Connection': 'close', 'x-compute-request-id': 'req-d813a2f5-fbfe-4eab-9eaf-5bb2ee36a635'}
RESP BODY: {"transfers": []}

[Eric Miller (erickmiller)]

I will try to rebuild everything from source this weekend and see if the problem still occurs when using the latest Kolla Ansible.

[Akihiro Motoki (amotoki)]

I tried this using the apache based deployment by DevStack. I am not sure what is different between DevStack apache deployments and kolla ansible deployments.

[Eric Miller (erickmiller)]

I ran out of time this weekend while working on an Octavia issue, so will try to reproduce the problem today built using the "source" option when building the images.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Dashboard (Horizon) because there has been no activity for 60 days.]