climate devstack rejoin climate-m error

[Dina Belova]

Now there is the following error using ./rejoin-stack.sh script for climate-m service:

user@climate:~/devstack$ cd /opt/stack/climate && /usr/local/bin/climate-manager --debug --config-file /etc/climate/climate.conf
2014-02-19 19:21:36.592 9506 DEBUG climate.openstack.common.lockutils [-] Got semaphore "dbapi_backend" lock /opt/stack/climate/climate/openstack/common/lockutils.py:170
2014-02-19 19:21:36.636 9506 DEBUG climate.openstack.common.lockutils [-] Got semaphore / lock "__get_backend" inner /opt/stack/climate/climate/openstack/common/lockutils.py:250
2014-02-19 19:21:38.649 9506 DEBUG climate.openstack.common.lockutils [-] Semaphore / lock released "__get_backend" inner /opt/stack/climate/climate/openstack/common/lockutils.py:254
2014-02-19 19:21:41.103 9506 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-02-19 19:21:41.124 9506 CRITICAL climate [-] HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 111] ECONNREFUSED)

That's because Keystone is not started yet, so trying to use it ends up with this error, it should be some kind of delay for it.

[Pablo Fernando Cargnelutti (pablo-fernando-cargnelutti)]

The one trying to authenticate against Keystone is get_aggregate_from_name_or_id in reservation_pool.py which is used by host_plugin during its setup.
Referring to the comment in line 60 of reservation_pool.py "use client wrapped by climate..." I will implement a client nova wrapper with the only addition of a configurable retry policy to fix this issue.

Comments? Ideas?

[Dina Belova (dbelova)]

Pablo, sorry, can't get your point. There is change by Swann removing direct assignments to novaclient, etc - https://bugs.launchpad.net/climate/+bug/1285584

This bug is about using devstack+climate

While ./rejoin-stack.sh script running there is error in climate-m tab, because the time climate-manager is trying to start, there is no keystone running. There should be some kind of sleep of smth like that on rejoin - to let keystone start in proper way before climate-manager

[Pablo Fernando Cargnelutti (pablo-fernando-cargnelutti)]

Dina, the thing is that the component of climate manager trying to use keystone is the setup method of the host_plugin. The suggestion is to add a simple wrapper to nova client which will retry a couple (configurable) times until fail. This should give time for keystone to start.
Other option is to just add a piece of code in the method that make the first call to keystone trying to delay it a few seconds. I have this second option working but I'm not sure if the cleanest implementation.

Besides cmd/manager, Is there any specific piece of code that only runs on rejoin? if there is, the sleep should be placed there.

[Dina Belova (dbelova)]

Hm, I was thinking about some extra line in climate devstack related lines like bash's 'sleep 5' - in some appropriate place. But probably, your retrying idea is nice too. But in this case please take a look on Swann's change + existing nova wrapper in climate.utils.openstack.nova

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/79452

[Pablo Fernando Cargnelutti (pablo-fernando-cargnelutti)]

Proposed solution under review: https://review.openstack.org/79452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/80021
Committed: https://git.openstack.org/cgit/stackforge/climate/commit/?id=10f59f3afb0e3d474845919817f0ce9b3619ac2c
Submitter: Jenkins
Branch: master

commit 10f59f3afb0e3d474845919817f0ce9b3619ac2c
Author: Swann Croiset <email address hidden>
Date: Wed Mar 12 18:21:10 2014 +0100

    Lets the cloud administrator create the freepool

    And no more create it in plugin setup.

    Fixes bug #1282534

    Change-Id: I07b6bf562bba6728368d63707ffd9b2e906ab523