nova-compute can not regist resource_provider when nova-compute start before placement endpoint created

[zhang zhenzhong]

1.When nova-compute start,bug placement's endpoint is not ready, it get the keystone session whitout placement's endpoint, cause nova-comuter regist resource_provider in nova_api failed.
2.Then create placement's endpoint, but nova-compute can not regist resource_provider automatic because of the old keystone session. In this case, It must restart the nova-compute to regist the resource_provider

I think the keystone client session should be reload when regist resource_provider failed.

[Matt Riedemann (mriedem)]

Which release are you testing on? Newton or Ocata? Or master (pike)? Because we do continue to retry in the compute service to connect to placement even if the first attempt fails:

https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/client/report.py#L192

Or is the problem with the ksa session created here?

https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/client/report.py#L190

[zhang zhenzhong (zzzhang0118)]

we found it in Ocata release
if report_client can not get the endpoint at init
https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/client/report.py#L190

it always reach line 412 when every update_availbility_resource in compute serice
https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/client/report.py#L412

[zhang zhenzhong (zzzhang0118)]

Log is endpoint can not found
https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/client/report.py#L59

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/482222

[OpenStack Infra (hudson-openstack)]

Change abandoned by zhang zhenzhong (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/482222

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[Chris Dent (cdent)]

Turning back on the in progress and assignment. This got lost, probably when the change was split out from an unrelated change:

https://review.openstack.org/#/c/483460/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/493536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493536
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c6b0d8ff5a80bf87f49124d8d9e4621d157c51e1
Submitter: Jenkins
Branch: master

commit c6b0d8ff5a80bf87f49124d8d9e4621d157c51e1
Author: Chris Dent <email address hidden>
Date: Mon Aug 14 13:47:56 2017 +0100

    Reset client session when placement endpoint not found

    If the report client is able to access keystone to get a service
    catalog, but that catalog does not include a placement service (because
    it hasn't been added to the catalog yet), creating resource providers
    and other placement entities will fail. This is expected.

    What's not expected is that creating entities will continue to fail for
    quite some time, even if placement is added to the catalog. This is
    because the keystone session caches the service catalog for some amount
    of time.

    Therefore we need to create a new client session when EndpointNotFound
    happens. This has been added in this change by extracting creation of
    the report client's _client to its own method that we can call from the
    exception handler. The resource provider and aggregate maps are wiped at
    this time, to make sure we are starting from a clean slate. While this
    isn't likely to cause a problem in real life scenarios, in the manual
    testing I was doing it created issues.

    I've made the _client method synchronized so that in the unlikely event
    that the resource tracker is trying to do its update job while some
    other thing is happening, we won't waste the client. This may not be
    necessary, but probably doesn't harm anything.

    Change-Id: I02ac615dc26a4a0d1fd28a638f622777e41d14e4
    Co-Authored-By: zhangzhenzhong <email address hidden>
    Closes-Bug: #1697825

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/483460
Reason: This was fixed at https://review.opendev.org/#/c/493536/