test_get_service_by_service_and_host_name intermittently fails if it tries to use nova-conductor on a single node devstack

[Matt Riedemann]

By default, devstack runs in superconductor mode, meaning there are 2 conductor services, one is in the cell0 database and one is in the cell1 database. These get reported with the same service binary "nova-conductor" and on the same host (since it's single node devstack). This can cause failures in this test:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/admin/test_services.py#n61

Which randomly picks a service from the list and then does a query by hostname and binary, and if it picks nova-conductor, it can fail because it's expecting one but there can be two conductors in this type of deployment.

This test probably shouldn't even exist in tempest since it can be tested in the nova tree (and probably already is) and it's not an interop guideline since it's an admin-only API by default.

[Matt Riedemann (mriedem)]

http://logs.openstack.org/71/568271/2/check/tempest-full-py3/7142c54/testr_results.html.gz

ft1.2: tempest.api.compute.admin.test_services.ServicesAdminTestJSON.test_get_service_by_service_and_host_name[id-39397f6f-37b8-4234-8671-281e44c74025]testtools.testresult.real._StringException: pythonlogging:'': {{{
2018-05-17 08:26:51,821 28427 INFO [tempest.lib.common.rest_client] Request (ServicesAdminTestJSON:test_get_service_by_service_and_host_name): 200 GET https://10.208.194.113/compute/v2.1/os-services 0.051s
2018-05-17 08:26:51,822 28427 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'content-location': 'https://10.208.194.113/compute/v2.1/os-services', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'connection': 'close', 'date': 'Thu, 17 May 2018 08:26:51 GMT', 'content-type': 'application/json', 'status': '200', 'x-openstack-request-id': 'req-328531e8-4cee-4a2d-a8b0-5079f5337a67', 'server': 'Apache/2.4.18 (Ubuntu)', 'content-length': '1050', 'x-compute-request-id': 'req-328531e8-4cee-4a2d-a8b0-5079f5337a67'}
        Body: b'{"services": [{"updated_at": "2018-05-17T08:26:46.000000", "binary": "nova-conductor", "zone": "internal", "state": "up", "id": 1, "status": "enabled", "disabled_reason": null, "host": "ubuntu-xenial-rax-iad-0004065631"}, {"updated_at": "2018-05-17T08:26:51.000000", "binary": "nova-compute", "zone": "nova", "state": "up", "id": 2, "status": "enabled", "disabled_reason": null, "host": "ubuntu-xenial-rax-iad-0004065631"}, {"updated_at": "2018-05-17T08:26:51.000000", "binary": "nova-scheduler", "zone": "internal", "state": "up", "id": 2, "status": "enabled", "disabled_reason": null, "host": "ubuntu-xenial-rax-iad-0004065631"}, {"updated_at": "2018-05-17T08:26:42.000000", "binary": "nova-consoleauth", "zone": "internal", "state": "up", "id": 5, "status": "enabled", "disabled_reason": null, "host": "ubuntu-xenial-rax-iad-0004065631"}, {"updated_at": "2018-05-17T08:26:45.000000", "binary": "nova-conductor", "zone": "internal", "state": "up", "id": 6, "status": "enabled", "disabled_reason": null, "host": "ubuntu-xenial-rax-iad-0004065631"}]}'
2018-05-17 08:26:51,913 28427 INFO [tempest.lib.common.rest_client] Request (ServicesAdminTestJSON:test_get_service_by_service_and_host_name): 200 GET https://10.208.194.113/compute/v2.1/os-services?binary=nova-conductor&host=ubuntu-xenial-rax-iad-0004065631 0.088s
2018-05-17 08:26:51,913 28427 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'content-location': 'https://10.208.194.113/compute/v2.1/os-services?binary=nova-conductor&host=ubuntu-xenial-rax-iad-0004065631', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'connection': 'close', 'date': 'Thu, 17 May 2018 08:26:51 GMT', 'content-type': ...

[Matt Riedemann (mriedem)]

This is pretty close to in-tree testing, except it doesn't filter on host:

https://github.com/openstack/nova/blob/4b4e005c639204f2c680eda61131a6930379d70e/nova/tests/functional/wsgi/test_services.py#L34

But we could update it to do that.

[Matt Riedemann (mriedem)]

There are a few functional tests in nova like this that list services filtering on host and binary:

https://github.com/openstack/nova/blob/4b4e005c639204f2c680eda61131a6930379d70e/nova/tests/functional/test_servers.py#L2210

So we should already have this covered.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/569112

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569112
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=6d593b981779dffd4ea74ea577c3f5170f14c878
Submitter: Zuul
Branch: master

commit 6d593b981779dffd4ea74ea577c3f5170f14c878
Author: Matt Riedemann <email address hidden>
Date: Thu May 17 09:26:46 2018 -0400

    Remove test_get_service_by_service_and_host_name

    By default, devstack runs in superconductor mode, meaning there
    are 2 conductor services, one is in the cell0 database and one
    is in the cell1 database. These get reported with the same service
    binary "nova-conductor" and on the same host (since it's single
    node devstack). This can cause failures in this test which
    randomly picks a service from the list and then does a query by
    hostname and binary, and if it picks nova-conductor, it can fail
    because it's expecting one but there can be two conductors in this
    type of deployment.

    This test shouldn't even exist in tempest since it can be (and is)
    tested in the nova tree and it's not an interop guideline since
    it's an admin-only API by default.

    Change-Id: I40a776959768f4422dd7960680a877db96cc3b9b
    Closes-Bug: #1771817

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 19.0.0 release.