Unable to create trust errors in glance-api

[Rajiv Mucheli]

Hi,

I enabled swift_store_expire_soon_interval = 1800 for images taking long time to complete the upload, but it doesnt seem to work as planned, i see a trust issue (based on docu, this is True by default) but i see the below in the glance-api logs :

021-02-18 14:04:52,948.948 30 INFO glance.api.v2.image_data [req-bb9660d8-c24c-4350-9d4e-7cfaffebf8d9 332d21d621e27dd887ff1f3388312be975597e42b755eec00ceff70d033228b8 97bf741678d44e8da33c43f4c4662ade - ec213443e8834473b579f7bea9e8c194 ec213443e8834473b579f7bea9e8c194] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token

2021-02-18 12:02:43,166.166 33 INFO glance.api.v2.image_data [req-8a48bfa1-9d37-4095-8f7f-70438d4daff6 a10475412aa34d05a815fac977df8620 caa6209d2c38450f8266311fd0f05446 - default 582d6603e91d4d3d8193fa9160a599f0] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.

http://paste.openstack.org/show/802787/

Ref :

https://bugs.launchpad.net/keystone/+bug/1775140
https://review.opendev.org/c/openstack/glance/+/479047

Glance Version : Victoria Release, 21.0.0
Glance_Store Version : 2.3.0
Swift Version (Backend) : Victoria (i have enabled multi-tenant)
Keystone : Train

Please let me know if further information is required.

Regards,
Rajiv

[Brian Rosmaita (brian-rosmaita)]

Could you try this with the log level set at debug? I can't find that error message in keystoneauth1 or glance_store, so I'm not sure where it's coming from. glance_store doesn't try to use collect_timing directly, so it must be somewhere else.

[Oliver (ciwi)]

I'm seeing the same messages on OpenStack Victoria 22.1.0. I set log level to debug, here's my output:

Apr 06 15:04:54 infra1-glance-container-99614ac2 glance-wsgi-api[1824]: 2021-04-06 15:04:54.274 1824 INFO glance.api.v2.image_data [req-c572bb10-58e1-4006-8f0e-9bd721b2986b 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.
Apr 06 15:07:14 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:10.195 1837 ERROR keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Bad response code while validating token: 500 An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672): keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672)
Apr 06 15:07:40 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:38.197 1837 WARNING keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Identity response: {"error": {"code": 500, "title": "Internal Server Error", "message": "An unexpected error prevented the server from fulfilling your request."}}: keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672)
Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: DAMN ! worker 13 (pid: 1836) died, killed by signal 9 :( trying respawn ...
Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: Respawned uWSGI worker 13 (new pid: 5799)
Apr 06 15:07:44 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:40.427 1837 CRITICAL keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to validate token: Failed to fetch token data from identity server: keystonemiddleware.auth_token._exceptions.ServiceError: Failed to fetch token data from identity server
Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1837]: Tue Apr 6 15:07:45 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during GET /v2/images?limit=1000&sort_key=created_at&sort_dir=desc (192.168.110.250)
Apr 06 15:07:45 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:45.629 1837 CRITICAL glance [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error
                                                                        2021-04-06 15:07:45.629 1837 ERROR glance OSError: write error
                                                                        2021-04-06 15:07:45.629 1...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/795772

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance/+/795772
Committed: https://opendev.org/openstack/glance/commit/9b683678b2a19bb1069aace58e054150a78b5f6f
Submitter: "Zuul (22348)"
Branch: master

commit 9b683678b2a19bb1069aace58e054150a78b5f6f
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/glance/+/800101

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/glance/+/800102

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/glance/+/800103

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 23.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance/+/800101
Committed: https://opendev.org/openstack/glance/commit/7ce475ce11ebb5813ce487bfa9cf64458ac08ac5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7ce475ce11ebb5813ce487bfa9cf64458ac08ac5
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052
    (cherry picked from commit 9b683678b2a19bb1069aace58e054150a78b5f6f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance/+/800102
Committed: https://opendev.org/openstack/glance/commit/3139ef39cb34c411d340572b098a790c92cfdf7e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 3139ef39cb34c411d340572b098a790c92cfdf7e
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052
    (cherry picked from commit 9b683678b2a19bb1069aace58e054150a78b5f6f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance/+/800103
Committed: https://opendev.org/openstack/glance/commit/f213c1bf44388fde5802bd707cf2850f88a26d5e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f213c1bf44388fde5802bd707cf2850f88a26d5e
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    merge-conflict on requirements jsonschema.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052
    (cherry picked from commit 9b683678b2a19bb1069aace58e054150a78b5f6f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 20.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 21.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 22.1.0 release.

[Wassilij Kaiser (wkaiser2023)]

Hello,

I upgraded from Victoria to Yoga.

DISTRIB_ID="OSA"
DISTRIB_RELEASE="25.2.0"
DISTRIB_CODENAME="Yoga"
DISTRIB_DESCRIPTION="OpenStack-Ansible"

I have the same error infra1-glance-container-dc13a04b glance-wsgi-api[282311]: 2023-01-23 13:25:06.745 282311 INFO glance.api.v2.image_data [req-10359154-0be4-4da1-9e85-2d94079c17b4 b2dca74976034d5b9925bdcb03470603 021ce436ab004cde851055bac66370bc - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.