default paste_deploy.flavor is none, but config file text implies it is 'keystone' (was: non-admin users can see all tenants' images even when image is private)

[Jonathan Mills]

[root@vm013 glance]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@vm013 glance]# rpm -qa |grep glance |sort
openstack-glance-16.0.1-1.el7.noarch
openstack-glance-doc-16.0.1-1.el7.noarch
python2-glanceclient-2.10.0-1.el7.noarch
python2-glance-store-0.23.0-1.el7.noarch
python-glance-16.0.1-1.el7.noarch
python-glanceclient-doc-2.10.0-1.el7.noarch
[root@vm013 glance]# md5sum /etc/glance/policy.json
a4f29d0f75bbc04f1d83a1abdf0fda6f /etc/glance/policy.json

I am running only Glance v2 API.

In this demo, as an un-privileged user, I will list all glance images, from all tenants, and they are all marked 'private'.

(as admin):
[root@vm013 ~]# openstack role assignment list --effective --names |grep jonathan
| user | jonathan@Default | | ozoneaq@ndc | | False |

(as jonathan):
[root@vm013 ~]# . keystonerc_jonathan
[root@vm013 ~]# printenv |grep OS_ |sort
OS_AUTH_URL=https://keystone.gpcprod:5000/v3
OS_CACERT=/etc/openldap/cacerts/gpcprod_root_ca.pem
OS_IDENTITY_API_VERSION=3
OS_PASSWORD=XXXXXXXXXXXXXXXXXX
OS_PROJECT_DOMAIN_NAME=NDC
OS_PROJECT_NAME=ozoneaq
OS_USER_DOMAIN_NAME=Default
OS_USERNAME=jonathan
OS_VOLUME_API_VERSION=3

[root@vm013 ~]# openstack image list
+--------------------------------------+-----------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+-----------------------------------+--------+
| 0099a343-1376-49f4-85f9-795624fb2ce8 | CentOS-7-x86_64-GenericCloud-1808 | active |
| 53d7c007-318b-4dad-b7cb-38b1dd31f884 | Ubuntu1604-180919 | active |
| 482f52ca-e56c-4555-a0e3-93eb491db389 | Ubuntu1604-20181016 | active |
| 212aaf3c-18f6-4327-8a11-c726c2e21780 | Ubuntu1804-20181016 | active |
| 051d2fff-6b90-4321-9c64-c613f0ddf3da | Windows2016Std-20181003r4 | active |
| ac6baa7c-fd2f-48e2-84e0-37a86f623e38 | Windows2016std-20181003r2 | active |
| 2264c6b9-40e7-492d-a5bc-dd11a7b4ee10 | Windows2016std-20181004 | active |
| 6d865748-ae7a-4c43-9d01-bc35c9002fd9 | Windows2016std-20181004r2 | active |
| 26ba1766-aa67-4b1b-81cd-90dda8d41384 | WindowsServer2016-20180926 | active |
| 3fc3c155-c7a2-4556-a5d0-de7eff208d7d | WindowsStd2016-20181010 | active |
| b6d161ca-e03b-46c5-95a0-5fe31723c5c7 | centos7-201810100 | active |
| 8bdc33be-1eb5-429b-b0ca-682b24df45f0 | centos7-gi-build-test1 | active |
| 34a915b8-cca6-45c3-9348-5e15dace444f | cirros | active |
| 84102d5c-1641-47bb-b727-a59e707e871c | keyshotslave-1604-snap2 | active |
| cedf9ae7-6adc-44d4-b7cb-d5664ea3fef0 | keyshotslave1604-snap1 | active |
| be4dbd67-d56f-41dd-8378-8aa6ca064f55 | mm-cirros-test | active |
| be67cf99-b545-4a91-a3d8-fe9f26a8854d | mm-cirros-test2 | active |
| a8dfd028-5911-4178-a77d-bb3da8996372 | mm-test-image4 | active |
| b6d9d44d-2e3c-48a9-9bf5-b6fca20979f9 | testt2-snap | active |
| 1c401eea-0e6e-475b-9a46-ffbfb388ca35 | ubuntu1804-180919 | active |
+--------------------------------------+-----------------------------------+--------+
[root@vm013 ~]# openstack image show cirros
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | 443b7623e27ecf03dc9e01ee93f67afe |
| container_format | bare |
| created_at | 2018-09-17T13:43:13Z |
| disk_format | raw |
| file | /v2/images/34a915b8-cca6-45c3-9348-5e15dace444f/file |
| id | 34a915b8-cca6-45c3-9348-5e15dace444f |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros |
| owner | 6e6d8ff081014c679f18ad4b818ffd4c |
| properties | direct_url='file:///var/lib/glance/images/34a915b8-cca6-45c3-9348-5e15dace444f', locations='[{u'url': u'file:///var/lib/glance/images/34a915b8-cca6-45c3-9348-5e15dace444f', u'metadata': {u'mountpoint': u'/var/lib/glance/images', u'type': u'nfs', u'id': u'gpc-b32-na-01', u'share_location': u'nfs://gpc-b32-na-01/glance'}}]' |
| protected | False |
| schema | /v2/schemas/image |
| size | 12716032 |
| status | active |
| tags | |
| updated_at | 2018-09-17T13:49:18Z |
| virtual_size | None |
| visibility | private |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

So you can see that my un-privileged user jonathan (role:user) just displayed the private image 'cirros' from tenant 6e6d8ff081014c679f18ad4b818ffd4c. User 'jonathan' is not a member of that tenant.

(as admin):
[root@vm013 ~]# openstack project list |grep 6e6d8ff081014c679f18ad4b818ffd4c
| 6e6d8ff081014c679f18ad4b818ffd4c | gpcadm |

Perhaps even stranger, as my admin user (role:admin, in admin tenant), I cannot set the visibility of an image to 'public':

[root@vm013 ~]# openstack image set --public cirros
403 Forbidden: You are not authorized to complete publicize_image action. (HTTP 403)

My /etc/glance/policy.json is identical to the reference one, here: https://raw.githubusercontent.com/openstack/glance/master/etc/policy.json

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Jonathan Mills (jonmills-t)]

This issue was posted to openstack-operators, where some responders said they were unable to duplicate the problem, and that's a big concern of mine. If it is a misconfiguration of my cluster, I still want to understand what went wrong and how to fix it, because this is a pretty serious problem. To shed a little light on our installation procedure, what we do is we provision our bare metal using xCAT, after which we apply RPM packages from the CentOS Cloud repo (http://mirror.centos.org/centos/7/cloud/x86_64/openstack-queens/) and configure services strictly according to the openstack docs (e.g. https://docs.openstack.org/glance/queens/install/install-rdo.html). We tweak the config files to use HTTPS and speak to our endpoints, etc, but there is really nothing exotic about our setup. We were running glance under uwsgi, behind apache+mod_proxy, but as part of our troubleshooting we reverted to using the openstack-glance-api systemd unit file that launches the old python2 eventlet -- there was no change in the exhibited behavior though.

This is our /etc/glance/glance-api.conf:

[DEFAULT]
show_image_direct_url = true
show_multiple_locations = true
location_strategy = store_type
enable_v1_api = false
enable_v2_api = true
enable_v1_registry = false
image_cache_dir = /var/lib/glance/images/image-cache
node_staging_uri = file:///var/lib/glance/images/tmp
use_user_token = true
auth_url = https://keystone.gpcprod:5000/v3
bind_host = 10.2.3.123
bind_port = 9292
workers = 6
http_keepalive = true
client_socket_timeout = 900
backlog = 4096
cert_file = /etc/glance/ssl/glance.pem
key_file = /etc/glance/ssl/glance.key
debug = true
log_file = /var/log/glance/api.log
log_dir = /var/log/glance
use_syslog = False
publish_errors = true
syslog_log_facility = LOG_USER
transport_url = rabbit://user@<email address hidden>:5671,user:<email address hidden>:5671,user:<email address hidden>:5671/

[cors]

[database]
connection = mysql+pymysql://glance:<email address hidden>/glance
connection_recycle_time=200

[glance_store]
stores = file,http
default_store = file
https_ca_certificates_file = /etc/openldap/cacerts/gpcprod_root_ca.pem
https_insecure = false
cinder_catalog_info = volumev3:cinderv3:publicURL
cinder_os_region_name = RegionOne
cinder_ca_certificates_file = /etc/openldap/cacerts/gpcprod_root_ca.pem
cinder_api_insecure = false
filesystem_store_metadata_file = /etc/glance/metadata.json
filesystem_store_datadir = /var/lib/glance/images
filesystem_store_file_perm = 0644
os_region_name=RegionOne

[image_format]
container_formats = ami,ari,aki,bare,ovf,ova
disk_formats = ami,ari,aki,vhd,vmdk,raw,qcow2,vdi,iso

[keystone_authtoken]
auth_uri = https://keystone.gpcprod:5000/v3
auth_url = https://keystone.gpcprod:5000/v3
cafile = /etc/openldap/cacerts/gpcprod_root_ca.pem
insecure = false
region_name = RegionOne
memcached_servers = 10.2.3.117:11211,10.2.3.118:11211,10.2.3.119:11211
token_cache_time = 1800
revocation_cache_time = 1800
memcache_security_strategy = None
memcache_pool_dead_retry = 300
memcache_pool_socket_timeout = 3
auth_type=password
username = glance
password = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
project_domain_name = default
user...

[Jonathan Mills (jonmills-t)]

Okay, we believe we understand the problem now.

According to the documentation (https://docs.openstack.org/glance/queens/configuration/glance_api.html), 'keystone' is the Default flavor of the [paste_deploy] section. In most other cases I've seen, when something is a default, it is safe to leave an INI section blank.

In our Queens (and Pike) configurations, we had a blank [paste_deploy] INI section. But we noticed that, in our older Mitaka rack, [paste_deploy] contained 'flavor = keystone'. After adding that value to [paste_deploy] in Queens, immediately image visibility works as expected.

Nevertheless, we still consider this a bug, either in the code or in the documentation. The docs must either insist that the use fill this in (or likely break the anticipated, normal behavior of Glance, and probably NOT say that Keystone is default); or the code needs to change to really use keystone as default in glance-api-paste.ini independent of any INI block or user intervention.

Thanks,

Jonathan

[Michael Moore (mdmoore2)]

correct patch file for https://github.com/openstack/glance/pull/9

[Jeremy Stanley (fungi)]

Since this bug is already public, related patches can be pushed directly to review.openstack.org (and it's a trivial enough patch that if someone wants to do that for Michael I don't think we need to worry about ICLA coverage for it, but make sure to git-am it so his authorship is correctly preserved).

[Michael Moore (mdmoore2)]

@ Jeremey I have just figured out how to deal with gerrit and believe it's submitted.

https://review.openstack.org/613671

[Brian Rosmaita (brian-rosmaita)]

Change is still under review.

[Brian Rosmaita (brian-rosmaita)]

Adjusted the bug title to reduce the freakout effect.

[Brian Rosmaita (brian-rosmaita)]

The problem is that the way a "sample_default" value is displayed (as opposed to an actual "default" value) is misleading. This has been fixed in oslo.config master (fix not yet released):
https://review.openstack.org/#/c/607353/

[Brian Rosmaita (brian-rosmaita)]

The oslo.config bug for this is https://bugs.launchpad.net/oslo.config/+bug/1795729

[Brian Rosmaita (brian-rosmaita)]

We use sample_default for several config values, so regardless of what's decided about changing the default value for paste_deploy.flavor, part of the fix for this bug should be a patch to update requirements.txt and lower-constraints.txt in glance once oslo.config 6.6.3 is released and upper-constraints.txt has been updated in the requirements repo.

[Brian Rosmaita (brian-rosmaita)]

I was having serious deja vu yesterday when looking at this. This is a duplicate of https://bugs.launchpad.net/glance/+bug/1711117 . Will mark the old one as a duplicate of this (which is a bit weird, but looks like the discussion on this bug explains that it's actually sort of fixed now).

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Moore (<email address hidden>) on branch: master
Review: https://review.opendev.org/613671
Reason: as mentioned by Brian, resolved in https://review.openstack.org/#/c/607353/

[Erno Kuvaja (jokke)]

The fix for this was released back in 2018 it seems. Closing the bug.

[Erno Kuvaja (jokke)]

For what it's worth I don't think advisory at this point is any use for anyone.

[Jeremy Stanley (fungi)]

Yes, since this bug is only valid for branches which are no longer in a maintained state, there is little point in issuing an advisory.