nova service-list for new compute service causes TypeError with servicegroup_driver mc

[s10]

Description
===========
nova service-list for new compute service causes TypeError.
Related commit: https://git.openstack.org/cgit/openstack/nova/commit/?id=0df91a7f799060cd2e9b8a0adac1efacb974bcb3

Steps to reproduce
==================
1. Set servicegroup_driver=mc and enable_new_services=False in /etc/nova/nova.conf
2. Add new compute host, start nova-compute service. Service becomes disabled, and field updated_at is NULL in nova.services table in DB.
3. Execute nova service-list.

Expected result
===============
List of the nova services

Actual result
=============
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-cbd9f4ca-7634-4378-8dcc-5a03d9d4193f)

Environment
===========
OpenStack Pike

Logs
==============
Trace in the nova-api-os-compute logs:

ERROR nova.api.openstack.extensions [req-6034517c-bc29-4ebe-931e-8726fd934bee e05fb82b34cd4265a839f2482debb973 b5d263d6d7c24b84b335f4f00ae6d7c9] Unexpected exception in API method: TypeError: can't compare datetime.datetime to NoneType
TRACE nova.api.openstack.extensions Traceback (most recent call last):
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 336, in wrapped
TRACE nova.api.openstack.extensions return f(*args, **kwargs)
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 261, in index
TRACE nova.api.openstack.extensions _services = self._get_services_list(req, ['forced_down'])
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 115, in _get_services_list
TRACE nova.api.openstack.extensions for svc in _services]
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 83, in _get_service_detail
TRACE nova.api.openstack.extensions updated_time = self.servicegroup_api.get_updated_time(svc)
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/servicegroup/api.py", line 84, in get_updated_time
TRACE nova.api.openstack.extensions return self._driver.updated_time(member)
TRACE nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/mc.py", line 81, in updated_time
TRACE nova.api.openstack.extensions if updated_time_in_db <= updated_time_in_mc:
TRACE nova.api.openstack.extensions TypeError: can't compare datetime.datetime to NoneType
TRACE nova.api.openstack.extensions
2018-07-16 11:40:16,437.437 2096 INFO nova.api.openstack.wsgi [req-6034517c-bc29-4ebe-931e-8726fd934bee e05fb82b34cd4265a839f2482debb973 b5d263d6d7c24b84b335f4f00ae6d7c9] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

[Matt Riedemann (mriedem)]

Looks like the correct analysis of the bug. We don't test the MC server group driver or enable_new_services config option very well, or together (obviously).

[Matt Riedemann (mriedem)]

It should be pretty easy to recreate this in a test which uses the MC driver and something like the oslo_cache.dict or some dogpile.cache.null backend.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/583927

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/583936

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/583927
Reason: This type of functional test won't work with the oslo_cache.dict since the cache will be a dict member variable of the server group API driver, and there are different server group API instances between the compute service and the API.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583936
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0f4e5c442c0ab446274979dafda873c1e1c20585
Submitter: Zuul
Branch: master

commit 0f4e5c442c0ab446274979dafda873c1e1c20585
Author: Kevin_Zheng <email address hidden>
Date: Thu Jul 19 19:09:31 2018 +0800

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/592310

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/592311

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/592337

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/592310
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=161d38679fec13bdb10e89536f7e0db43176396f
Submitter: Zuul
Branch: stable/rocky

commit 161d38679fec13bdb10e89536f7e0db43176396f
Author: Kevin_Zheng <email address hidden>
Date: Thu Jul 19 19:09:31 2018 +0800

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f
    (cherry picked from commit 0f4e5c442c0ab446274979dafda873c1e1c20585)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/592311
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3b17251da6e06260ee4b499dd18b46cc95f97fe5
Submitter: Zuul
Branch: stable/queens

commit 3b17251da6e06260ee4b499dd18b46cc95f97fe5
Author: Kevin_Zheng <email address hidden>
Date: Thu Jul 19 19:09:31 2018 +0800

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f
    (cherry picked from commit 0f4e5c442c0ab446274979dafda873c1e1c20585)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/592337
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b9ea6939db92662f4978866e9d48c43e67568922
Submitter: Zuul
Branch: stable/pike

commit b9ea6939db92662f4978866e9d48c43e67568922
Author: Vlad Gusev <email address hidden>
Date: Thu Aug 16 13:14:01 2018 +0300

    Fix service list for disabled compute using MC driver

    In 0df91a7f799060cd2e9b8a0adac1efacb974bcb3 a comparison
    between ``updated_at`` field from services table and update time
    from memcache was added if the servergroup_driver is set to be
    mc.

    This could be problematic if servergroup_driver=mc and
    ``[DEFAULT]/enable_new_services`` option in /etc/nova/nova.conf
    is set to be ``False`` at the same time, as the ``updated_at``
    field will always be ``Null`` in services record.

    This patch fixes the bug by returning ``updated_time_in_mc`` instead
    under the above mentioned scenario.

    Closes-Bug: #1781880

    Conflict is caused by I10b84a9cc7f0cc8ea213a377e4c4f16168b6e45b
    not being in Pike.

    Change-Id: I27160140cbccf3e6914aed2dcff54d547e446c1f
    (Cherry-picked from 0f4e5c442c0ab446274979dafda873c1e1c20585)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.