Container security policy fetch prevents creating container when using ceph

[Daniel Queen]

Starting in Ussuri release (horizon 18.3.0), horizon tries to pull a list of security policies from swift so you can choose during creation. If you are using ceph as your object storage, this check fails to get the list of policies and the fetch fails and prevents you from creating any containers. Containers and objects can still be manipulated via CLI.

Dashboard error message is thrown upon clicking Object-Store > Containers dashboard.. "Error: Unable to fetch the policy details." You can see existing containers, but unable to create any new ones. Security Policy isn't fetched and is required to create one in horizon.

Basically it seems with the new feature in horizon, it totally breaks the dashboard functionality if you are using ceph as your backend.

environment details:
CentOS 8.1
Openstack Ussuri
openstack-dashboard-18.3.2-1.el8.noarch
Ceph Octopus 15.2.2

apparent culprit: https://opendev.org/openstack/horizon/commit/b62c49acf5eba001937cff33a41d73a4f4caa292

[Daniel Queen (crono782)]

Did some more research on this. It isn't strictly a bug in horizon, but rather a shortfall in swiftclient. Swiftclient isn't capable of reading capabilities from ceph and cannot read policies by default. Modifying /usr/lib/python3.6/swiftclient/client.py to use the modified /swift/info url that ceph provides and restarting httpd and memcached fixes the issue. Without this modification the new horizon feature still blocks creation of new containers by default. Either swiftclient needs fixing to allow for ceph schema or horizon needs an option to ignore this (prefer swiftclient feature tbh)

[Akihiro Motoki (amotoki)]

horizon primarily assumes the swift API from the same release (so horizon ussuri assumes swift ussuri).
It looks like this bug is caused by a mismatch in supported feature in Swift and Ceph.

The first option would be that Ceph provides supports a list of security policies.
The second option would be to add a workaround in horizon. Hopefully someone can implement it.

Considering the current development resource of horizon, it is not realistic to cover feature gaps between Swift and its compatible API provided by Ceph, so we assume Ceph swift-compatible API provides all features in Swift when implementing new features. I think closing feature gaps is a responsibility of a service which provides a compatible API.

[Akihiro Motoki (amotoki)]

Marking it as Incomplete because it should be discussed as a feature gap in Ceph.

[Daniel Queen (crono782)]

To clarify, ceph does provide the feature, but it resides at a different url location than swift provides. Here is my workaround:
sed -i '/def get_capabilities(self/,/return/{s|'/info'|/swift/info|}' /usr/lib/python3.6/site-packages/swiftclient/client.py

Again, as it stands, with this new feature if you are using ceph as a swift backend, the UI is broken. This workaround fixes swiftclient and thus horizon.

[Akihiro Motoki (amotoki)]

According to comment #6, Ceph does not provide a compatible API. This leads to a question that how horizon can provide an enough support for such variants. Horizon team checks the swift API reference for a new feature, but it is hard to check such variants unless teams/projects who provide such variant API(s). Should the horizon team check variants? It is too hard. If so, such projects who provides compatible APIs should be aware of all changes related to swift in horizon and send workarounds. That's the reason I marked it as Incomplete.

[Akihiro Motoki (amotoki)]

correction: but it is hard to check such variants unless teams/projects who provide such variant API(s). -> but it is hard to check such variants unless teams/projects who provide such variant API(s) checks horizon changes.

[Akihiro Motoki (amotoki)]

s/According to comment #6/According to comment #4/
(I am not sure how I saw the wrong number....)

[Akihiro Motoki (amotoki)]

IMHO I am okay to review a patch as a workaround if someone would like to work on the patch, but note that we have no verification against Ceph deployment, so Ceph swift support (not Swift support) is experimental or best effort basis.

[Daniel Queen (crono782)]

I gotcha, makes sense. The fix really should come for swiftclient I reckon. It's probably better to say that horizon's swift UI isn't compatible w/ ceph backed swift until swiftclient works with ceph properly (or workaround used).

[Eric H (eric-a-h)]

I've used kolla-ansible to set up swift and get this same error. I am not using ceph.
Below is the error generated in the horizon.log

[Sat Jul 25 21:38:39.442267 2020] [wsgi:error] [pid 24:tid 140282269677312] [remote 192.168.123.11:37870] OPENSTACK_IMAGE_BACKEND has a format "" unsupported by glance
[Sat Jul 25 21:38:39.442605 2020] [wsgi:error] [pid 24:tid 140282269677312] [remote 192.168.123.11:37870] OPENSTACK_IMAGE_BACKEND has a format "docker" unsupported by glance
[Sat Jul 25 21:38:39.442678 2020] [wsgi:error] [pid 24:tid 140282269677312] [remote 192.168.123.11:37870] OPENSTACK_IMAGE_BACKEND has a format "ova" unsupported by glance
[Sat Jul 25 21:38:43.258656 2020] [wsgi:error] [pid 23:tid 140282269677312] [remote 192.168.123.11:38000] REQ: curl -i http://192.168.123.100:8080/info -X GET -H "Accept-Encoding: gzip"
[Sat Jul 25 21:38:43.258811 2020] [wsgi:error] [pid 23:tid 140282269677312] [remote 192.168.123.11:38000] RESP STATUS: 401 Unauthorized
[Sat Jul 25 21:38:43.259011 2020] [wsgi:error] [pid 23:tid 140282269677312] [remote 192.168.123.11:38000] RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="http://192.168.123.100:5000"', 'X-Trans-Id': 'txef9b065378b042cd86937-005f1ca663', 'X-Openstack-Request-Id': 'txef9b065378b042cd86937-005f1ca663', 'Date': 'Sat, 25 Jul 2020 21:38:43 GMT'}
[Sat Jul 25 21:38:43.259176 2020] [wsgi:error] [pid 23:tid 140282269677312] [remote 192.168.123.11:38000] RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}'
[Sat Jul 25 21:38:43.260422 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] REQ: curl -i http://192.168.123.100:8080/info -X GET -H "Accept-Encoding: gzip"
[Sat Jul 25 21:38:43.260548 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] RESP STATUS: 401 Unauthorized
[Sat Jul 25 21:38:43.260753 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="http://192.168.123.100:5000"', 'X-Trans-Id': 'tx8a8d059901764c9db88f3-005f1ca663', 'X-Openstack-Request-Id': 'tx8a8d059901764c9db88f3-005f1ca663', 'Date': 'Sat, 25 Jul 2020 21:38:43 GMT'}
[Sat Jul 25 21:38:43.260879 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}'
[Sat Jul 25 21:38:43.261258 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] error invoking apiclient
[Sat Jul 25 21:38:43.261285 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] Traceback (most recent call last):
[Sat Jul 25 21:38:43.261297 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] File "/usr/share/openstack-dashboard/openstack_dashboard/api/rest/utils.py", line 128, in _wrapped
[Sat Jul 25 21:38:43.261308 2020] [wsgi:error] [pid 26:tid 140282269677312] [remote 192.168.123.11:38020] data = function(self, request, *args, *...

[Eric H (eric-a-h)]

Per a different bug thread, setting

#delay_auth_decision = False
delay_auth_decision = True

In swift-proxy-server gets this to work. Perhaps move my comment to kolla-ansible?

Thanks

[Donny Davis (donny-g)]

following

[Tobias Urdin (tobias-urdin)]

This should be solved by this https://bugs.launchpad.net/python-swiftclient/+bug/1712358 but maybe not released yet

[Akihiro Motoki (amotoki)]

python-swiftclient 3.10.0 release (victoria) includes the fix. The fix was backported to stable/ussuri python-swiftclient, but it was not released yet. If you need it for ussuri, use the latest commit of swiftclient at the moment.

[Akihiro Motoki (amotoki)]

bug 1712358 in python-swiftclient is a same issue. Marking this as duplicate.

[yehaifeng (yehaifeng)]

I got the same error at version wallaby installed by kolla-ansbile when I click Object Store -> Containers.

[Fri Jul 15 01:01:29.134308 2022] [wsgi:error] [pid 77:tid 140379401193216] [remote 192.168.115.7:46308] REQ: curl -i https://vip.test.com:8080/info -X GET -H "Accept-Encoding: gzip"
[Fri Jul 15 01:01:29.134501 2022] [wsgi:error] [pid 77:tid 140379401193216] [remote 192.168.115.7:46308] RESP STATUS: 401 Unauthorized
[Fri Jul 15 01:01:29.134698 2022] [wsgi:error] [pid 77:tid 140379401193216] [remote 192.168.115.7:46308] RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="https://vip.test.com:5000"', 'X-Trans-Id': 'tx51937d0917254696b6d65-0062d0bc69', 'X-Openstack-Request-Id': 'tx51937d0917254696b6d65-0062d0bc69', 'Date': 'Fri, 15 Jul 2022 01:01:29 GMT'}
[Fri Jul 15 01:01:29.134870 2022] [wsgi:error] [pid 77:tid 140379401193216] [remote 192.168.115.7:46308] RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}'
[Fri Jul 15 01:01:29.157363 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] REQ: curl -i https://vip.test.com:8080/info -X GET -H "Accept-Encoding: gzip"
[Fri Jul 15 01:01:29.157531 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] RESP STATUS: 401 Unauthorized
[Fri Jul 15 01:01:29.157753 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="https://vip.test.com:5000"', 'X-Trans-Id': 'tx12229c7102cf4141bc263-0062d0bc69', 'X-Openstack-Request-Id': 'tx12229c7102cf4141bc263-0062d0bc69', 'Date': 'Fri, 15 Jul 2022 01:01:29 GMT'}
[Fri Jul 15 01:01:29.157944 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}'
[Fri Jul 15 01:01:29.159174 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] error invoking apiclient
[Fri Jul 15 01:01:29.159205 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] Traceback (most recent calllast):
[Fri Jul 15 01:01:29.159215 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] File "/var/lib/kolla/venv/lib/python3.6/site-packages/openstack_dashboard/api/rest/utils.py", line 128, in _wrapped
[Fri Jul 15 01:01:29.159223 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] data = function(self, request, *args, **kw)
[Fri Jul 15 01:01:29.159231 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] File "/var/lib/kolla/venv/lib/python3.6/site-packages/openstack_dashboard/api/rest/swift.py", line 53, in get
[Fri Jul 15 01:01:29.159238 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] policies = capabilities['swift']['policies']
[Fri Jul 15 01:01:29.159284 2022] [wsgi:error] [pid 80:tid 140379401193216] [remote 192.168.115.7:46714] KeyError: 'swift'
[Fri Jul 15 01:01:29.159330 2022] [wsgi:err...