Timing issue in remote-keystone deployment model.

[Le Tian Ren]

If keystone service is deployed in another node separately from other OpenStack api services like glance, neutron, we define this as remote-keystone mode, if they are all in the same node, we define this as local-keystone mode.

This bug is much easier to reproduce in remote-keystone mode when an OpenStack api services node is rebooted, all the api services will automatically start as planned in /etc/rc3.d/, this takes some time before all the services are ready. At the same time our powervc-driver sync services in the same node also start automatically after reboot, since the keystone server is in another node, which means it doesn't need restart, then our client factory's initialize_local_servicecatalog doesn't need to wait for keystone being ready as it does in local-keystone mode, which means initialize_local_servicecatalog does discover and build endpoint services earlier than it does in local-keystone mode, as long as an api service are not ready when it is being discovered, the endpoints list will not be filled with the real versions, e.g. network or image (v1 are hardcoded by default)

2014-08-18 05:46:00.730 2601 INFO powervc.common.client.factory [-] {u'compute': {u'v2': [<powervc.common.client.service.NovaService object at 0x1ebadd0>]}, u'network': {'v1': [<powervc.common.client.service.NeutronService object at 0x21f9410>]}, u'image': {'v1': [<powervc.common.client.service.GlanceService object at 0x22d8250>]}, u'metering': {'v1': []}, u'cloudformation': {u'v1': []}, u'volume': {u'v1': [<powervc.common.client.service.CinderService object at 0x22d8690>]}, u'orchestration': {u'v1': []}, u'identity': {u'v3': [<powervc.common.client.service.KeystoneService object at 0x22d86d0>]}}

Then when the corresponding sync service call the missing version api, the NoneType(NULL) client will report errors like.

2014-08-18 05:46:13.892 2601 WARNING powervc.glance.manager.manager [-] An error occurred during startup image synchronization: 'NoneType' object has no attribute 'images'
2014-08-18 05:46:13.892 2601 INFO powervc.glance.manager.manager [-] Startup image synchronization did not complete successfully. It will run again in 60 seconds.

Then if manually restart the sync serivce after a while when all OpenStack api services are ready, initialize_local_servicecatalog can discover and build endpoint services with correct versions like this

2014-08-18 06:11:24.274 4626 INFO powervc.common.client.factory [-] {u'compute': {u'v2': [<powervc.common.client.service.NovaService object at 0x3665dd0>]}, u'network': {u'v2.0': [<powervc.common.client.service.NeutronService object at 0x39a4590>]}, u'image': {u'v1.1': [<powervc.common.client.service.GlanceService object at 0x3a852d0>], u'v2.2': [<powervc.common.client.service.GlanceService object at 0x3a85690>]}, u'metering': {'v1': []}, u'cloudformation': {u'v1': []}, u'volume': {u'v1': [<powervc.common.client.service.CinderService object at 0x36658d0>]}, u'orchestration': {u'v1': []}, u'identity': {u'v3': [<powervc.common.client.service.KeystoneService object at 0x3a856d0>]}}

Then everything goes fine.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115519

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115519
Committed: https://git.openstack.org/cgit/stackforge/powervc-driver/commit/?id=52947efccd8541831728575fe873b84273aa97f3
Submitter: Jenkins
Branch: master

commit 52947efccd8541831728575fe873b84273aa97f3
Author: Le Tian Ren <email address hidden>
Date: Wed Aug 20 15:05:35 2014 +0800

    Timing issue in remote-keystone deployment model

    This is actually timing issue between openstack service endpoints becoming
    active and powervc-driver's client initialization of those.

    1. get_client() call with version_filter might runs into the 'None' client
    error mentioned in the bug, which means *currently* only glance sync is
    affected. When no version filtered, we do a rediscover for the svc_type.

    2. get_client() call without version_filter or new_client() call just choose
    the latest avaiable version service, so no 'None' client error, although the
    chosen version might not be the real *latest* but the default hardcoded 'v1'
    version, due to the timing issue mentioned in the bug. For the latter case,
    unless there is an observation/notification mechanism or poll mechanism for
    service versions update, based on current design, there is no perfect solution
    as far as I can imagine.

    Change-Id: I9ca0f906c8954ecd621be22445c12fd6a55a1ae7
    Closes-Bug: 1358215