Volume hosts test does not check service type

[Lucian Petrut]

The VolumeHostsAdminTestJSON.test_show_host test will randomly choose one of the available cinder hosts and query the api for host details.

The issue is that this test does not check the retrieved service type and will fail if the randomly selected host does not run a cinder-volume service.

https://github.com/openstack/tempest/blob/44c2e0f24a2a7e781c711fbefdd177a004f9c029/tempest/api/volume/admin/test_volume_hosts.py#L53-L54

Trace: http://paste.openstack.org/raw/609691/

[Jeremy Zhang (jeremy.zhang)]

Hi, Lucian.
Firstly I want to know whether you ran the tests in Mitaka or prior?
Now the Mitaka is EOL in master branch of the Tempest.

[Lucian Petrut (petrutlucian94)]

Hi Jeremy,

We had this issue recently on our CI (running on master branch), had to disable the test.

To give you a better picture: we have a single c-vol service, running on Windows. When the test randomly selects the node hosting the Cinder scheduler service, the 'host show' operation fails, as there's no c-vol service running on that host.

CI results sample:
http://64.119.130.115/cinder/463989/1/iscsi/tempest/results.html.gz
http://paste.openstack.org/show/610180/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477366

[Ghanshyam Mann (ghanshyammann)]

Yes, i can verify the behaviour on my multinode env.

[root@indctrtg01 ~]# curl -g -i -X GET http://indctrtg01:8776/v2/37f71a6f300d4191a77bc135de496ef4/os-hosts -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABZlDVy2rs_hrEGC5krT9wFZNMPDZboxBRHFehP4Kb7pfX1z2fGggQx7ed0rBmTPjEyWL2kwFXbOjlNgr6TJYXsczFrpO90yIiHcfQqjVRkEkNJLmRTMeiTIYr4-fo1Bsq5XYD3t1kZm7nI2QG-T2RfM_OLDVGzpWrfvrb2ELCw-EqnaIQ"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-c60baf15-6ff1-492e-850d-cdb9e8de821b
Content-Type: application/json
Content-Length: 780
X-Openstack-Request-Id: req-c60baf15-6ff1-492e-850d-cdb9e8de821b
Date: Wed, 16 Aug 2017 12:11:59 GMT

{"hosts": [{"service-status": "available", "service": "cinder-scheduler", "zone": "nova", "service-state": "enabled", "host_name": "indctrtg01", "last-update": "2017-08-16T12:11:50.000000"}, {"service-status": "unavailable", "service": "cinder-volume", "zone": "nova", "service-state": "enabled", "host_name": "WIN-GN8FHSVQRAQ@myWindowsiSCSIBackend", "last-update": "2017-08-16T19:08:38.000000"}, {"service-status": "unavailable", "service": "cinder-volume", "zone": "nova", "service-state": "enabled", "host_name": "WIN-SVGBAT9OOD9@myWindowsiSCSIBackend", "last-update": "2017-08-17T12:10:52.000000"}, {"service-status": "available", "service": "cinder-backup", "zone": "nova", "service-state": "enabled", "host_name": "indctrtg01", "last-update": "2017-08-16T12:11:50.000000"}]}[root@indctrtg01 ~]#
[root@indctrtg01 ~]#
[root@indctrtg01 ~]#

GET host not running cinder-volume service. (cinder-scheduler service running on this host). 404 strange.

[root@indctrtg01 ~]# curl -g -i -X GET http://indctrtg01:8776/v2/37f71a6f300d4191a77bc135de496ef4/os-hosts/indctrtg01 -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABZlDVy2rs_hrEGC5krT9wFZNMPDZboxBRHFehP4Kb7pfX1z2fGggQx7ed0rBmTPjEyWL2kwFXbOjlNgr6TJYXsczFrpO90yIiHcfQqjVRkEkNJLmRTMeiTIYr4-fo1Bsq5XYD3t1kZm7nI2QG-T2RfM_OLDVGzpWrfvrb2ELCw-EqnaIQ"
HTTP/1.1 404 Not Found
Content-Length: 106
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-4024b61a-01ac-40e5-b3b4-2ccb36714ed5
X-Openstack-Request-Id: req-4024b61a-01ac-40e5-b3b4-2ccb36714ed5
Date: Wed, 16 Aug 2017 12:12:50 GMT

{"itemNotFound": {"message": "Service cinder-volume could not be found on host indctrtg01.", "code": 404}}[root@indctrtg01 ~]#

GET host running cinder-volume service. SUCCESS

[root@indctrtg01 ~]# curl -g -i -X GET http://indctrtg01:8776/v2/37f71a6f300d4191a77bc135de496ef4/os-hosts/WIN-GN8FHSVQRAQ@myWindowsiSCSIBackend -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABZlDVy2rs_hrEGC5krT9wFZNMPDZboxBRHFehP4Kb7pfX1z2fGggQx7ed0rBmTPjEyWL2kwFXbOjlNgr6TJYXsczFrpO90yIiHcfQqjVRkEkNJLmRTMeiTIYr4-fo1Bsq5XYD3t1kZm7nI2QG-T2RfM_OLDVGzpWrfvrb2ELCw-EqnaIQ"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-3b0533d3-b0f8-4660-8e04-3b59ef09478b
Content-Type: application/json
Content-Length: 397
X-Openstack-Request-Id: req-3b0533d3-b0f8-4660-8e04-3b59ef09478b
Date: Wed, 16 Aug 2017 12:13:51 GMT

{"host": [{"resource": {"volume_count": "5", "total_volume_gb": "6", "total_snapshot_gb": "0", "project": "(total)", "...

[Ghanshyam Mann (ghanshyammann)]

This is what cinder code is written but i am not sure if it is right and we are missing doc only. API ref does not say that it get host of cinder-volume service only.

cinder API get host with service cinder-volume
https://github.com/openstack/cinder/blob/6dd6c74b389f15c42d3158ad50493c1d3086452f/cinder/api/contrib/hosts.py#L153

topic is cinder-volume here - https://github.com/openstack/cinder/blob/6dd6c74b389f15c42d3158ad50493c1d3086452f/cinder/common/constants.py#L26

DB raise 404 if no cinder-volume service running -

https://github.com/openstack/cinder/blob/6dd6c74b389f15c42d3158ad50493c1d3086452f/cinder/objects/service.py#L125
https://github.com/openstack/cinder/blob/6dd6c74b389f15c42d3158ad50493c1d3086452f/cinder/db/sqlalchemy/api.py#L537

[Ghanshyam Mann (ghanshyammann)]

Adding cinder also to confirm this and if this is right behaviour then at least update the doc.

[Sean McGinnis (sean-mcginnis)]

Looks like a valid bug in Cinder. That should not be restricted to the cinder-volume service.

[Sean McGinnis (sean-mcginnis)]

Looking more, it looks like getting a specific host has always been meant as a way to get the volume usage stats for a given cinder-volume host. It's been there for as far back as I see.

That does make the API a little schizophrenic. A list of all hosts returns everything, but you can only get the one specific type of host.

Even though I find it a little odd (and just feels wrong to me) since it is long established behavior, I will update the api-ref to call that out for the GET on a specific host.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477366
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=af55ae3b47e5b535b404267f4ce96b0dec237951
Submitter: Jenkins
Branch: master

commit af55ae3b47e5b535b404267f4ce96b0dec237951
Author: jeremy.zhang <[e-mail]>
Date: Fri Aug 18 11:36:46 2017 +0800

    Fix the test case for showing host

    From the implementation of Cinder show host API, we can know that the API
    is to show volume usage info on the specified cinder-volume host, which is
    not fully described in official api-ref. If the specified host does not run
    the cinder-volume service, or the cinder-volume service is disabled on the
    host, the API should fail. This patch aims to solve this problem.

    Change-Id: I40e64c52a5a71d3a7fd7e7f04b86b6c35d49594e
    Closes-Bug: #1691144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499285
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=86a96625d74015d10b179f5f5f34b2940814b62e
Submitter: Jenkins
Branch: master

commit 86a96625d74015d10b179f5f5f34b2940814b62e
Author: Sean McGinnis <email address hidden>
Date: Wed Aug 30 14:59:11 2017 -0500

    api-ref: Clarify os-host GET behavior

    Getting all hosts returns every Cinder service host, regardless
    of which service is running on the host. But getting a specific
    host will fail if you try it for a host not running cinder-volume.

    This looks to be the behavior from the beginning, but there was
    nothing denoting this, causing errors and confusion for code that
    thought it could get a host from the list, then get detailed info
    about that host. The details return volume and snapshot counts,
    so it really only makes sense for cinder-volume, so the code ends
    up returning a 404 for anything else.

    This API design seems a little disjointed, but since this is how
    it appears to have always been, just make sure the api docs have
    the right details for potential API consumers to know what to
    expect.

    Change-Id: If53279cfcbbde1297bb2e55e17d17b473e7d0d6e
    Closes-bug: #1691144

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 17.1.0 release.