cinder service isn't working via horizon after doing a Rocky->Stein->Train upgrade

[xinliang]

Description:

cinder service isn't working via horizon after doing a Rocky->Stein->Train upgrade.
We use kollan-ansible to do the upgrade. And lvm as volume backend.

Horizon dashboard http://HORIZON-DASHBOARD-HOST/project/volumes/ page display empty and just says:
"Error: Unable to retrieve volume list."
"Error: Unable to retrieve snapshot list."

whereas it is working via openstack client tool, say "openstack volume list" output is ok.

Other info:
=====> cinder-api-access.log via horizon <=======
- - - [21/Feb/2020:01:44:25 +0000] "GET /v1/3f1cbbce59704574b423b5baaafaaae0 HTTP/1.1" 404 154 2068 "-" "python-requests/2.22.0"
- - - [21/Feb/2020:01:44:25 +0000] "GET /v1/3f1cbbce59704574b423b5baaafaaae0 HTTP/1.1" 404 154 1926 "-" "python-requests/2.22.0"

=====> cinder-api-access.log via openstack client <=======
- - - [21/Feb/2020:01:43:40 +0000] "GET /v3/3f1cbbce59704574b423b5baaafaaae0/volumes/detail HTTP/1.1" 200 3321 90044 "-" "python-cinderclient"

[xinliang (xin3liang)]

Tried several times Rocky->Stein->Train upgrade to confirm this issue is existed.

After upgrading:
linaro@j12-d05:~/liuxl/kolla-ansible$ openstack service list --long
+----------------------------------+-------------+----------------+----------------------------------------+---------+
| ID | Name | Type | Description | Enabled |
+----------------------------------+-------------+----------------+----------------------------------------+---------+
| 03ade6e515b74c9192c5f7efb318e77b | neutron | network | Openstack Networking | True |
| 042822fda67b4456be2c3ef5b8837221 | cinder | volume | Openstack Block Storage | True |
| 148f0aeb3b4c4fc49c0a722fc1a16d76 | nova_legacy | compute_legacy | OpenStack Compute Service (Legacy 2.0) | True |
| 4e513cdd8c194206bb2cb8aa22dfcf97 | placement | placement | Placement Service | True |
| 60b68ca6713e4a7ea01bd3a8e94db344 | heat-cfn | cloudformation | Orchestration | True |
| 7ed1a7fd2ef2461f89be2042a7650da8 | keystone | identity | Openstack Identity Service | True |
| cbeaf0d5fafc435aa97b4f721c62645e | cinderv3 | volumev3 | Openstack Block Storage | True |
| cf2ab1b0734b422496fb08ed55753b6c | glance | image | Openstack Image | True |
| d75854e0745844ae9b710c0f994c80dd | cinderv2 | volumev2 | Openstack Block Storage | True |
| e1d3159728bb4c6f9080b2c9bc9583be | nova | compute | OpenStack Compute Service | True |
| f8211687216f424bb13e3995ce597e3c | heat | orchestration | Orchestration | True |
+----------------------------------+-------------+----------------+----------------------------------------+---------+
linaro@j12-d05:~/liuxl/kolla-ansible$ openstack volume service list --long
+------------------+---------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated At | Disabled Reason |
+------------------+---------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | j12-d05 | nova | enabled | up | 2020-02-21T02:50:34.000000 | None |
| cinder-volume | j12-d05@lvm-1 | nova | enabled | up | 2020-02-21T02:50:38.000000 | None |
+------------------+---------------+------+---------+-------+----------------------------+-----------------+
linaro@j12-d05:~/liuxl/kolla-ansible$ openstack volume list
+--------------------------------------+------+--------+------+----------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------+--------+------+----------------------------------+
| 43341624-0ed3-40fc-aa41-08a38c9f975c | | in-use | 1 | Attached to ciros-2 on /dev/vda |
| 06719f60-e740-45e2-bf8e-7c1829ede0f6 | | in-use | 1...

[xinliang (xin3liang)]

Other info:
If I just do a Rocky->Stein or Stein->Train upgrade, this issue will not happen.

[xinliang (xin3liang)]

From above cinder-api-access.log, horizon is using an unavailable v1 api, don't know why it will use v1 api.

[xinliang (xin3liang)]

There are some getting cinder api version errors in horizon.log

[Radosław Piliszek (yoctozepto)]

What does catalog say?

[xinliang (xin3liang)]

The catalog list tells that there is a v1 cinder volume service. Because there is a V1 cinder volume service and endpoint see commet #1 log.

I am sure this v1 cinder volume service is created at rocky kolla-ansible deplyment. Because v1 cinder service is removed in later kolla-ansible version.

Then I delete v1 cinder volume service and endpoint manually and re-login horizon. The volumes page go back to work again.

By debugging horizon code, the reason might be that horizon use v1 cinder volume api to fetch version info, which might be removed in train cinder, then cinder return 404 response , then horizon can't handle this response and raise an exception making horizon stop working.

[Radosław Piliszek (yoctozepto)]

Bah, I'm bad at seeing "read more". :-)

So kolla-ansible should remove removed endpoints.
Otherwise the behavior is mostly undefined - may fail or not, depending on client interface.

Notifying horizon in case they want to strengthen this logic for future.

[xinliang (xin3liang)]

Yes, horizon need to be more strengthen.

Cinder client release notes tells it has remove v1 support:
https://docs.openstack.org/releasenotes/python-cinderclient/train.html#relnotes-5-0-0-stable-train

Horizon always get version info begin with v1 api:
https://github.com/openstack/horizon/blob/stable/train/openstack_dashboard/api/cinder.py#L269

Don't know if rocky kolla-ansible should remove cinder v1 api endpoint and service.

[Akihiro Motoki (amotoki)]

As a quick look from horizon perspective, the logic to check cinder microversion availability looks wrong. The default cinderclient instance is configured based on OPENSTACK_API_VERSION['cinder'] and service catalog returned from keystone, but the logic for microversion check does not honor this. This is the cause of the failure.

Note that horizon considers a case where 'volume' endpoint is used for Cinder v3 API, but it is a different story from this bug.

[Akihiro Motoki (amotoki)]

I mark this as High in horizon as the bug should happen when multiple cinder API versions are configured in the service catalog. It is not limited to upgrade cases.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/709417

[xinliang (xin3liang)]

Anyway, v1 api endpoint should be removed for kolla-ansible Rocky. Becuase it is removed completely at Queens: https://docs.openstack.org/releasenotes/cinder/queens.html#relnotes-12-0-0-stable-queens-upgrade-notes.
So cherry-pick the removal patch for kolla-ansible Rocky.

[Radosław Piliszek (yoctozepto)]

But we should amend it to actually *remove* that endpoint on any upgrade.
Odd that, despite being removed upstream, it actually still works for Horizon in Rocky and Stein...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709417
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=487a518d286957159e7beff854a15aae5b384a3b
Submitter: Zuul
Branch: stable/rocky

commit 487a518d286957159e7beff854a15aae5b384a3b
Author: wu.chunyang <[e-mail]>
Date: Mon Dec 24 14:39:43 2018 +0000

    remove cinder v1 endpoint

    now,cinder does not support v1 endpoint,so no need to create it in
    keystone

    (cherry picked from commit 4939fc28d2eeccb6da218375c74288234944ed60)

    Cinder v1 api is removed completely at Queens[1]. So its endpoint
    should be removed for kolla-ansible Rocky also.

    Related-Bug: 1864133

    Change-Id: I4b3bbf24c19a996e73f4b360beeb20a110907a6d
    [1]: https://docs.openstack.org/releasenotes/cinder/queens.html#relnotes-12-0-0-stable-queens-upgrade-notes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/709954

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709954
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=4b31ae5063e4392ecf964ffaae15a30bf97ddedc
Submitter: Zuul
Branch: master

commit 4b31ae5063e4392ecf964ffaae15a30bf97ddedc
Author: Akihiro Motoki <email address hidden>
Date: Wed Feb 26 02:26:29 2020 +0900

    Check volume endpoint availability in the same order

    cinder microversion check (api.cinder.get_microversion) checks
    volume endpoint availability in a different order as cinderclient()
    does. It does not API_VERSIONS setting in horizon.
    As a result, when multiple volume endpoints are configured,
    get_microversion() accesses a volume endpoint with a different API
    version. At the moment cinder v2 and v3 APIs returns the same info,
    so it only affects when cinder v1 endpoint is configured.

    This commit introduces a new function _find_cinder_url() to
    retrieve a volume endpoint considering API_VERSIONS.

    get_auth_params_from_request() is no longer needed and the variable
    substitutions are now backed to cinderclient(). It was introduced
    so that the memoized decorator worked but the memoized decorator
    was improved and we no longer need it.

    Change-Id: I69b1fc11caf8a78c49e98aba6c538e5f344b14f2
    Closes-Bug: #1864133

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/710732

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710732
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=9b54bad89965cec22afc004df2c5e5a888e3bd06
Submitter: Zuul
Branch: stable/train

commit 9b54bad89965cec22afc004df2c5e5a888e3bd06
Author: Akihiro Motoki <email address hidden>
Date: Wed Feb 26 02:26:29 2020 +0900

    Check volume endpoint availability in the same order

    cinder microversion check (api.cinder.get_microversion) checks
    volume endpoint availability in a different order as cinderclient()
    does. It does not API_VERSIONS setting in horizon.
    As a result, when multiple volume endpoints are configured,
    get_microversion() accesses a volume endpoint with a different API
    version. At the moment cinder v2 and v3 APIs returns the same info,
    so it only affects when cinder v1 endpoint is configured.

    This commit introduces a new function _find_cinder_url() to
    retrieve a volume endpoint considering API_VERSIONS.

    get_auth_params_from_request() is no longer needed and the variable
    substitutions are now backed to cinderclient(). It was introduced
    so that the memoized decorator worked but the memoized decorator
    was improved and we no longer need it.

    Change-Id: I69b1fc11caf8a78c49e98aba6c538e5f344b14f2
    Closes-Bug: #1864133
    (cherry picked from commit 4b31ae5063e4392ecf964ffaae15a30bf97ddedc)

[Mark Goddard (mgoddard)]

I'm also hitting this. Can we get a stable/train release of horizon including the fix please?