Zookeeper servicegroup driver crashes

[Roman Dobosz]

Zookeeper driver is based on zookeeper and evzookeeper modules. The latter
is the source of nasty crash, which is well visible on nova conductor. To
reproduce it is enough to enable zookeeper in nova.conf, provide
configuration for the zookeeper service address and stack the thing. The
traceback:

2015-04-14 13:23:22.622 TRACE nova Traceback (most recent call last):
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/bin/nova-conductor", line 10, in <module>
2015-04-14 13:23:22.622 TRACE nova sys.exit(main())
2015-04-14 13:23:22.622 TRACE nova File "/opt/stack/nova/nova/cmd/conductor.py", line 44, in main
2015-04-14 13:23:22.622 TRACE nova manager=CONF.conductor.manager)
2015-04-14 13:23:22.622 TRACE nova File "/opt/stack/nova/nova/service.py", line 277, in create
2015-04-14 13:23:22.622 TRACE nova db_allowed=db_allowed)
2015-04-14 13:23:22.622 TRACE nova File "/opt/stack/nova/nova/service.py", line 146, in __init__
2015-04-14 13:23:22.622 TRACE nova self.servicegroup_api = servicegroup.API(db_allowed=db_allowed)
2015-04-14 13:23:22.622 TRACE nova File "/opt/stack/nova/nova/servicegroup/api.py", line 76, in __init__
2015-04-14 13:23:22.622 TRACE nova *args, **kwargs)
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 38, in import_object
2015-04-14 13:23:22.622 TRACE nova return import_class(import_str)(*args, **kwargs)
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 27, in import_class
2015-04-14 13:23:22.622 TRACE nova __import__(mod_str)
2015-04-14 13:23:22.622 TRACE nova File "/opt/stack/nova/nova/servicegroup/drivers/zk.py", line 28, in <module>
2015-04-14 13:23:22.622 TRACE nova evzookeeper = importutils.try_import('evzookeeper')
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 71, in try_import
2015-04-14 13:23:22.622 TRACE nova return import_module(import_str)
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 57, in import_module
2015-04-14 13:23:22.622 TRACE nova __import__(import_str)
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/lib/python2.7/dist-packages/evzookeeper/__init__.py", line 26, in <module>
2015-04-14 13:23:22.622 TRACE nova from evzookeeper import utils
2015-04-14 13:23:22.622 TRACE nova File "/usr/local/lib/python2.7/dist-packages/evzookeeper/utils.py", line 26, in <module>
2015-04-14 13:23:22.622 TRACE nova class _SocketDuckForFdTimeout(greenio._SocketDuckForFd):
2015-04-14 13:23:22.622 TRACE nova AttributeError: 'module' object has no attribute '_SocketDuckForFd'

The root cause of the problem is the change, which have a place in module
eventlet 0.17, which evzookeeper module is depend on. Because of the change the way eventlet.greenio is exposed there is no way to reach the class
_SocketDuckForFd other way than explicitly importing it via
eventlet.greenio.py2 module.

Definitely solution for this problem might go upstream to the evzookeeper
author, however the development of evzookeeper module seems stalled (no
activity for last 2 years), maybe it's worth to consider changing the zk
driver implementation to use different zk module (kazoo seems quite active).

Downgrading eventlet was no option for me, since keystone requires version
>=0.17.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/246343

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/246343
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=337a1b029a1f144f577a78712413a4182dd525f8
Submitter: Jenkins
Branch: master

commit 337a1b029a1f144f577a78712413a4182dd525f8
Author: Mark McLoughlin <email address hidden>
Date: Tue Nov 17 10:17:44 2015 +0000

    servicegroup: remove the zookeeper driver

    We have had a "untested and risky to use in production" log warning
    message for this driver since Kilo, it is currently broken (see below),
    there are no obviously active users or contributors, and we are planning
    on enabling Zookeeper usage by adopting the tooz library.

    bug #1443910 shows that the driver fails to load because eventlet 0.17
    broke evzookeeper by moving _SocketDuckForFd from eventlet.greenio to
    eventlet.greenio.py2 in commit 449c90a. The 0.17 release was in Feb,
    2015. The evzookeeper library hasn't been updated since Sep 2012 and the
    sole maintainer is the original author of the zookeeper servicegroup
    driver.

    The tooz driver spec - Ibf70c2dbe308fc8e4dd277d8c75c4445b3dfce90 -
    proposes a formal deprecation period for the zk driver, during which
    existing zk driver users are encouraged to move to tooz. However, given
    the state of the zk driver, we must conclude that there are no existing
    users who need a graceful migration path. Removing the driver will
    avoid potential confusion for new users and simplify the path to
    adopting tooz.

    Closes-Bug: #1443910
    Closes-Bug: #1414517
    Closes-Bug: #1414536

    Signed-off-by: Mark McLoughlin <email address hidden>
    Change-Id: I2dba71e71b1ed7cf8476e8bfe9481e84be5df128

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.0.0.0b3 development milestone.