swift driver ignores user_domain_name and project_domain_name settings

[Dr. Jens Harbott]

It seems difficult to get complete data for non-voting tests, but all results that I have checked starting on 2016-08-04 show the same failure in the devstack setup phase while trying to upload the cirros image:

2016-08-19 10:18:25.627 21976 DEBUG keystoneclient.auth.identity.v3.base [req-5d46098e-a4e6-469e-968c-ff9269f1f05e 91bbd5bf421b448a966030ed9f2f80bf 77a269bd219646a0a370134a3150d970 - default default] Making authentication request to http://127.0.0.1/identity/v3/auth/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:189
2016-08-19 10:18:28.796 21976 DEBUG keystoneclient.session [req-5d46098e-a4e6-469e-968c-ff9269f1f05e 91bbd5bf421b448a966030ed9f2f80bf 77a269bd219646a0a370134a3150d970 - default default] Request returned failure status: 401 request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:418
2016-08-19 10:18:28.797 21976 DEBUG oslo_messaging._drivers.amqpdriver [req-5d46098e-a4e6-469e-968c-ff9269f1f05e 91bbd5bf421b448a966030ed9f2f80bf 77a269bd219646a0a370134a3150d970 - default default] CAST unique_id: b5337d20062a4b708029b33f2b7d0cc0 NOTIFY exchange 'glance' topic 'notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data [req-5d46098e-a4e6-469e-968c-ff9269f1f05e 91bbd5bf421b448a966030ed9f2f80bf 77a269bd219646a0a370134a3150d970 - default default] Failed to upload image data due to internal error
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data Traceback (most recent call last):
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 114, in upload
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data image.set_data(data, size)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/domain/proxy.py", line 195, in set_data
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data self.base.set_data(data, size)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/notifier.py", line 449, in set_data
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data _send_notification(notify_error, 'image.upload', msg)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data self.force_reraise()
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/notifier.py", line 396, in set_data
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data self.repo.set_data(data, size)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/api/policy.py", line 185, in set_data
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/quota/__init__.py", line 304, in set_data
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data self.image.set_data(data, size=size)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/opt/stack/new/glance/glance/location.py", line 430, in set_data
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data verifier=verifier)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 371, in add_to_backend
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data verifier)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 344, in store_add_to_backend
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data verifier=verifier)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data return store_op_fun(store, *args, **kwargs)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 662, in add
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data allow_reauth=need_chunks) as manager:
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 1301, in get_manager_for_store
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data store, store_location, context, allow_reauth)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/connection_manager.py", line 64, in __init__
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data self.storage_url = self._get_storage_url()
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/connection_manager.py", line 160, in _get_storage_url
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data raise exceptions.BackendException(msg)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data BackendException: Cannot find swift service endpoint : The request you have made requires authentication. (HTTP 401) (Request-ID: req-9dc24405-0c91-40b3-bbd0-284c4b12edbe)
2016-08-19 10:18:28.802 21976 ERROR glance.api.v2.image_data

I've tried to reproduce this error on locally running trusty instances using the reproduce.sh from various failures, but the failure does not happen for me there, everything is just working fine, so there seems to be some correlation to the infra trusty images.

[Dr. Jens Harbott (j-harbott)]

O.k., after finding a bug in devstack-gate when creating reproduce.sh, I can see the error now locally. It seems related to OSC getting confused when --os-cloud is used in combination with other variables being set (e.g. by sourcing openrc). Initially everything is fine:

stack@jr-t5:~/devstack$ openstack --os-cloud devstack token issue
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| expires | 2016-09-13 21:19:39.381346+00:00 |
| id | ffcc967f3ba34321a7e25a7d0b322ca6 |
| project_id | a8512069cebe4f929681504422a931cb |
| user_id | ec224c917f8641b18a6c163441dd8345 |
+------------+----------------------------------+
stack@jr-t5:~/devstack$ openstack --os-cloud devstack-admin token issue
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| expires | 2016-09-13 21:19:48.164681+00:00 |
| id | 660da9458ceb40cd8955c35c1bfe3cb7 |
| project_id | cb5b39acff6a49bc871cba73cd746578 |
| user_id | 0da7c28639cf4f0b9e8c60077e7a54d1 |
+------------+----------------------------------+

But after sourcing openrc things for the other cloud definition get off track:

stack@jr-t5:~/devstack$ . openrc
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
stack@jr-t5:~/devstack$ openstack --os-cloud devstack token issue
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| expires | 2016-09-13 21:19:59.915116+00:00 |
| id | 4491e058a9f440a88912a006af8e9f54 |
| project_id | a8512069cebe4f929681504422a931cb |
| user_id | ec224c917f8641b18a6c163441dd8345 |
+------------+----------------------------------+
stack@jr-t5:~/devstack$ openstack --os-cloud devstack-admin token issue
The request you have made requires authentication. (HTTP 401) (Request-ID: req-b8772035-9c99-4b85-8ed3-af1c8c4deeb3)

[Dr. Jens Harbott (j-harbott)]

O.k., disregard the previous comment. The culprit is https://review.openstack.org/297665, this sets 'default' as default value for user_domain_id and project_domain_id. The result of this is that glance always uses these values in its auth requests and pays not attention to user_domain_name and project_domain_name which are set in /etc/glance/glance-swift-store.conf. After reverting the above patch in the locally installed glance_store, everything seems to be working fine again.

[Nikhil Komawar (nikhil-komawar)]

We've narrowed down on the issue and in a gist, the default values for *_domain_id are clashing when only *_domain_name are being set in the config file http://logs.openstack.org/75/369675/3/check/gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv/f748e8b/logs/etc/glance/glance-swift-store.conf.txt.gz

""" (in case file gets deleted later)

[ref1]
auth_version = 3
project_domain_name = service
user_domain_name = service
auth_address = http://127.0.0.1/identity/v3
key = secretservice
user = service:glance-swift

"""

Conversation from wednesday sept 14th morning is at http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2016-09-14.log.html#t2016-09-14T12:21:32

[Nikhil Komawar (nikhil-komawar)]

The domain_id will not override the domain_name values and this needs further investigation for why we do see failures.

My current hunch says that the default values for domain_id are not working out so the authentication request is only partially correct (hence a 401).

The workflow is as follows:

1. First swift store driver checks for existence of "glance-swift-store.conf" file https://github.com/openstack/glance_store/blob/5736659fe0edb4c8fd4d583a77017dc8beba774b/glance_store/_drivers/swift/utils.py#L121

2. This being true for your case, _load_config is called https://github.com/openstack/glance_store/blob/5736659fe0edb4c8fd4d583a77017dc8beba774b/glance_store/_drivers/swift/utils.py#L147

3. Defaults are passed, when configs are read from file

https://github.com/openstack/glance_store/blob/5736659fe0edb4c8fd4d583a77017dc8beba774b/glance_store/_drivers/swift/utils.py#L151

and

https://github.com/openstack/glance_store/blob/5736659fe0edb4c8fd4d583a77017dc8beba774b/glance_store/_drivers/swift/utils.py#L100-L107

4. Config values are used here and then passed to ks client https://github.com/openstack/glance_store/blob/5736659fe0edb4c8fd4d583a77017dc8beba774b/glance_store/_drivers/swift/utils.py#L163-L173

[Nikhil Komawar (nikhil-komawar)]

A partial (temporary) fix for identity v3 (original issue reported scenario) is at https://review.openstack.org/#/c/369675/

I think that workaround makes sense as reverting the patch will break other team:

I was told that the initial request for defaults was from ansible team. So, if we remove them in a hurry it will destabilize given they must have had a setup accordingly.

However, I'm of opinion that the defaults values need to be something that all project can live with. So, we need to have some cross project convo with keystone (for what happens when *_id and *_name values clash -- best way to resolve), ansible and tempest v3 gate.

If possible, let's avoid defaults in the code and use them in the config file. That way given multiple accounts are allowed to be setup in the config file, individual teams are free to setup their account specific defaults in the file.

[Dr. Jens Harbott (j-harbott)]

The code in [1] uses user_domain_id if it is set, user_domain_name is only even looked at if user_domain_id == None.

So setting a default value for user_domain_id is bound to break things. Maybe an alternative solution for ansible might be to instead default to user_domain_name='default', assuming their default domain is called that way.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identity/v3/password.py#L43

[Nikhil Komawar (nikhil-komawar)]

I agree with Dr. Jens' comment #6. We need to revert the change https://review.openstack.org/297665 and either the ansible team needs to setup defaults they need or we could potentially add another account ( [ref2] ) in the "glance-swift-store.conf" file for ansible team that will have domain_id as 'default' by default.

[Nikhil Komawar (nikhil-komawar)]

Revert for https://review.openstack.org/#/c/297665/
proposed here https://review.openstack.org/373438

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nikhil Komawar (<email address hidden>) on branch: master
Review: https://review.openstack.org/373438
Reason: As per Kairat's comments, I agree that we'd go with some sort of deprecation.

So, we will favor https://review.openstack.org/#/c/373726/ with deprecation path.

[Ian Cordasco (icordasc)]

Hoping that this will be picked back up for Pike.

[Ian Cordasco (icordasc)]

Also, if this is done well, we can backport this to Ocata. Whomever picks up Kairat's change should really keep the stable guidelines in mind.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kairat Kushaev (<email address hidden>) on branch: master
Review: https://review.openstack.org/373726

[Daniel 'f0o' Preussker (dpreussker)]

This still affects Rocky.

Adding *_domain_id entries into the swift config solved it.

Any ETA on the fixes?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/815009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/815009
Committed: https://opendev.org/openstack/glance_store/commit/6738d0b156fc1caae0604eb7bd8f05a6ed1c2d68
Submitter: "Zuul (22348)"
Branch: master

commit 6738d0b156fc1caae0604eb7bd8f05a6ed1c2d68
Author: Takashi Kajinami <email address hidden>
Date: Thu Oct 21 22:57:52 2021 +0900

    Swift: Honor *_domain_name parameters

    The *_domain_id parmaeters should not have any default. Otherwise
    keystoneauth ignores the *_domain_name parameters and it requires
    only *_domain_id parameters are used.

    Closes-Bug: #1620999
    Change-Id: I1f8c9184761313f9fc5fda2f257e52233e0196d1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 4.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/glance_store/+/854467

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/854467
Committed: https://opendev.org/openstack/glance_store/commit/4e5b90a4d200316492dff2c00ff0fe530f9025f8
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 4e5b90a4d200316492dff2c00ff0fe530f9025f8
Author: Takashi Kajinami <email address hidden>
Date: Thu Oct 21 22:57:52 2021 +0900

    Swift: Honor *_domain_name parameters

    The *_domain_id parmaeters should not have any default. Otherwise
    keystoneauth ignores the *_domain_name parameters and it requires
    only *_domain_id parameters are used.

    Conflicts:
            glance_store/_drivers/swift/utils.py

    Closes-Bug: #1620999
    Change-Id: I1f8c9184761313f9fc5fda2f257e52233e0196d1
    (cherry picked from commit 6738d0b156fc1caae0604eb7bd8f05a6ed1c2d68)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 3.0.1 release.