test_os_service_disable_enable fails with "ERROR (NotFound): Could not find binary nova-scheduler on host"

[Matt Riedemann]

Seen here:

http://logs.openstack.org/41/435141/19/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/894e11f/console.html#_2017-06-24_07_44_54_947034

2017-06-24 07:44:54.919101 | 2017-06-24 07:44:54.918 | {0} novaclient.tests.functional.v2.test_os_services.TestOsServicesNovaClientV211.test_os_service_disable_enable [1.927162s] ... FAILED
2017-06-24 07:44:54.920189 | 2017-06-24 07:44:54.919 |
2017-06-24 07:44:54.921445 | 2017-06-24 07:44:54.921 | Captured traceback:
2017-06-24 07:44:54.923085 | 2017-06-24 07:44:54.922 | ~~~~~~~~~~~~~~~~~~~
2017-06-24 07:44:54.924463 | 2017-06-24 07:44:54.924 | Traceback (most recent call last):
2017-06-24 07:44:54.925897 | 2017-06-24 07:44:54.925 | File "novaclient/tests/functional/v2/legacy/test_os_services.py", line 36, in test_os_service_disable_enable
2017-06-24 07:44:54.927180 | 2017-06-24 07:44:54.926 | service = self.nova('service-disable %s %s' % (host, serv.binary))
2017-06-24 07:44:54.928693 | 2017-06-24 07:44:54.928 | File "novaclient/tests/functional/base.py", line 314, in nova
2017-06-24 07:44:54.930090 | 2017-06-24 07:44:54.929 | endpoint_type, merge_stderr)
2017-06-24 07:44:54.932181 | 2017-06-24 07:44:54.931 | File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 130, in nova
2017-06-24 07:44:54.933322 | 2017-06-24 07:44:54.933 | 'nova', action, flags, params, fail_ok, merge_stderr)
2017-06-24 07:44:54.934956 | 2017-06-24 07:44:54.934 | File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 374, in cmd_with_auth
2017-06-24 07:44:54.936084 | 2017-06-24 07:44:54.935 | self.cli_dir, prefix=self.prefix)
2017-06-24 07:44:54.937305 | 2017-06-24 07:44:54.937 | File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 71, in execute
2017-06-24 07:44:54.938430 | 2017-06-24 07:44:54.938 | result_err)
2017-06-24 07:44:54.939609 | 2017-06-24 07:44:54.939 | tempest.lib.exceptions.CommandFailed: Command '['/opt/stack/new/python-novaclient/.tox/functional/bin/nova', '--os-username', 'admin', '--os-tenant-name', 'admin', '--os-password', 'secretadmin', '--os-auth-url', 'http://10.11.235.59/identity', '--os-compute-api-version', '2.11', '--os-endpoint-type', 'publicURL', 'service-disable', 'ubuntu-xenial-osic-cloud1-s3500-9498382', 'nova-scheduler']' returned non-zero exit status 1.
2017-06-24 07:44:54.941114 | 2017-06-24 07:44:54.940 | stdout:
2017-06-24 07:44:54.943100 | 2017-06-24 07:44:54.942 |
2017-06-24 07:44:54.945711 | 2017-06-24 07:44:54.943 | stderr:
2017-06-24 07:44:54.947034 | 2017-06-24 07:44:54.946 | ERROR (NotFound): Could not find binary nova-scheduler on host ubuntu-xenial-osic-cloud1-s3500-9498382. (HTTP 404) (Request-ID: req-540e1c6a-f5b2-4d9d-be81-fe5c047a72b1)

This is due to changes in the compute API making it impossible to disable non-nova-compute services since there is no HostMapping for them, and the API looks up the host via host mapping to find which cell it's in.

I'm actually surprised this isn't 100% failure:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%20(NotFound)%3A%20Could%20not%20find%20binary%20nova-scheduler%20on%20host%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

[Matt Riedemann (mriedem)]

This is the change in the API causing the issue: https://review.openstack.org/#/c/442162/

[Matt Riedemann (mriedem)]

This was also mentioned in the openstack-dev and openstack-operators mailing lists:

http://lists.openstack.org/pipermail/openstack-dev/2017-June/118308.html

I've added nova to this bug since we should have a release note for the upgrade impact here. There isn't anything in the Pike series release notes for this issue yet.

https://docs.openstack.org/releasenotes/nova/unreleased.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477212
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=e7b84daa8774434f3b1382ce40741bebe248de12
Submitter: Jenkins
Branch: master

commit e7b84daa8774434f3b1382ce40741bebe248de12
Author: Matt Riedemann <email address hidden>
Date: Sun Jun 25 09:43:12 2017 -0400

    Deprecate binary argument in nova service enable/disable/force-down CLIs

    Change If1e03c9343b8cc9c34bd51c2b4d25acdb21131ff in the API in Pike
    makes the os-services API look up services records via the host mapping
    in the API datadabase to determine which cell the service lives in.
    The host mappings only exist for nova-compute services, which means
    you can only enable/disable/force-down nova-compute services now, which
    is realistically the only service that ever made sense for those actions.

    That change broke some functional tests in novaclient though which loop
    through all services and disables them or forces them down. This change
    fixes those tests by only attempting the action on nova-compute services.

    This change also deprecates the binary argument to the service
    enable/disable/force-down commands since the only value that ever really
    worked or does anything is nova-compute, so a future version of the CLI
    should just hard-code that value.

    Change-Id: Idd0d2be960ca0ed59097c10c931da47a1a3e66fb
    Closes-Bug: #1700359

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-novaclient 9.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/489621

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/489621
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=598314a4bb53ffff2a8872bd6b290ef8d1d6c532
Submitter: Jenkins
Branch: master

commit 598314a4bb53ffff2a8872bd6b290ef8d1d6c532
Author: Matt Riedemann <email address hidden>
Date: Tue Aug 1 10:21:43 2017 -0400

    Add release note for PUT /os-services/* for non-compute services

    Change If1e03c9343b8cc9c34bd51c2b4d25acdb21131ff made the
    "PUT /os-services/*" APIs only work with nova-compute services
    since they now rely on finding the compute service in a cell
    via the host_mappings record in the API database, and host
    mappings only exist for compute services.

    No one should have relied on disabling a non-compute service since
    that does nothing, at least with in-tree code.

    This change just adds a release note for awareness.

    Change-Id: Idb5bbc31a0aabc328f11e5a9979e5b80651a8f67
    Closes-Bug: #1700359

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0rc1 release candidate.