Devstack nova-compute fails with ZeroMQ

[Oleksii Zamiatin]

Deploy devstack with ZeroMQ:

git clone https://github.com/openstack-dev/devstack.git
cd devstack

Copy local.conf from samples

Add this line to local.conf (select zeromq for messaging):
ENABLED_SERVICES+=,-rabbit,-qpid,zeromq,n-cpu,n-net

Run deployment:
./stack.sh

Deployment fails with error code.

Run ./rejoin-stack.sh

Go to nova-compute screen, see:

2014-11-24 11:34:04.079 CRITICAL nova [req-1369c998-f2ef-476f-b1b4-f808ee2f6b73 None None] ImportError: No module named rpc.matchmaker_redis

2014-11-24 11:34:04.079 TRACE nova Traceback (most recent call last):
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/bin/nova-compute", line 10, in <module>
2014-11-24 11:34:04.079 TRACE nova sys.exit(main())
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/cmd/compute.py", line 72, in main
2014-11-24 11:34:04.079 TRACE nova db_allowed=CONF.conductor.use_local)
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/service.py", line 275, in create
2014-11-24 11:34:04.079 TRACE nova db_allowed=db_allowed)
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/service.py", line 157, in __init__
2014-11-24 11:34:04.079 TRACE nova self.conductor_api.wait_until_ready(context.get_admin_context())
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/conductor/api.py", line 313, in wait_until_ready
2014-11-24 11:34:04.079 TRACE nova timeout=timeout)
2014-11-24 11:34:04.079 TRACE nova File "/opt/stack/nova/nova/baserpc.py", line 62, in ping
2014-11-24 11:34:04.079 TRACE nova return cctxt.call(context, 'ping', arg=arg_p)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-11-24 11:34:04.079 TRACE nova retry=self.retry)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-11-24 11:34:04.079 TRACE nova timeout=timeout, retry=retry)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 954, in send
2014-11-24 11:34:04.079 TRACE nova return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 945, in _send
2014-11-24 11:34:04.079 TRACE nova allowed_remote_exmods=self._allowed_remote_exmods)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 736, in _multi_send
2014-11-24 11:34:04.079 TRACE nova queues = _get_matchmaker().queues(topic)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py", line 832, in _get_matchmaker
2014-11-24 11:34:04.079 TRACE nova matchmaker = importutils.import_object(mm, *args, **kwargs)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/utils/importutils.py", line 38, in import_object
2014-11-24 11:34:04.079 TRACE nova return import_class(import_str)(*args, **kwargs)
2014-11-24 11:34:04.079 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo/utils/importutils.py", line 27, in import_class
2014-11-24 11:34:04.079 TRACE nova __import__(mod_str)
2014-11-24 11:34:04.079 TRACE nova ImportError: No module named rpc.matchmaker_redis
2014-11-24 11:34:04.079 TRACE nova
n-cpu failed to start

[Li Ma (nick-ma-z)]

It seems wrong configuration is set in devstack scripts.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/143293

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/143293
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=ce1524d0843894b099c4abd20a64f92acad50c55
Submitter: Jenkins
Branch: master

commit ce1524d0843894b099c4abd20a64f92acad50c55
Author: Li Ma <email address hidden>
Date: Sun Dec 21 00:46:34 2014 -0800

    Fix proper oslo.messaging object for zeromq driver

    Currently, as almost all the rpc driver of projects
    have been switched to oslo.messaging, the object should
    be imported via oslo.messaging rather than its own rpc lib.

    Change-Id: I9633446e78cb5af21f61a26f6fb365a8ed57a85f
    Partially-Implements: blueprint zeromq
    Closes-Bug: #1395721

[Oleksii Zamiatin (ozamiatin)]

Redis is now ok, in nova have:
2015-02-05 17:25:37.348 TRACE nova Traceback (most recent call last):
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/bin/nova-network", line 9, in <module>
2015-02-05 17:25:37.348 TRACE nova load_entry_point('nova==2015.1.dev837', 'console_scripts', 'nova-network')()
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/cmd/network.py", line 72, in main
2015-02-05 17:25:37.348 TRACE nova db_allowed=CONF.conductor.use_local)
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/service.py", line 275, in create
2015-02-05 17:25:37.348 TRACE nova db_allowed=db_allowed)
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/service.py", line 157, in __init__
2015-02-05 17:25:37.348 TRACE nova self.conductor_api.wait_until_ready(context.get_admin_context())
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/conductor/api.py", line 315, in wait_until_ready
2015-02-05 17:25:37.348 TRACE nova timeout=timeout)
2015-02-05 17:25:37.348 TRACE nova File "/opt/stack/nova/nova/baserpc.py", line 62, in ping
2015-02-05 17:25:37.348 TRACE nova return cctxt.call(context, 'ping', arg=arg_p)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-02-05 17:25:37.348 TRACE nova retry=self.retry)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-02-05 17:25:37.348 TRACE nova timeout=timeout, retry=retry)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 926, in send
2015-02-05 17:25:37.348 TRACE nova return self._send(target, ctxt, message, wait_for_reply, timeout)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 917, in _send
2015-02-05 17:25:37.348 TRACE nova allowed_remote_exmods=self._allowed_remote_exmods)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 794, in _multi_send
2015-02-05 17:25:37.348 TRACE nova envelope, allowed_remote_exmods)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 726, in _call
2015-02-05 17:25:37.348 TRACE nova msg = msg_waiter.recv()
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_zmq.py", line 228, in recv
2015-02-05 17:25:37.348 TRACE nova return self.sock.recv_multipart(**kwargs)
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python2.7/dist-packages/eventlet/green/zmq.py", line 359, in recv_multipart
2015-02-05 17:25:37.348 TRACE nova return _Socket_recv_multipart(self, flags, copy, track)
2015-02-05 17:25:37.348 TRACE nova File "/usr/lib/python2.7/dist-packages/zmq/sugar/socket.py", line 265, in recv_multipart
2015-02-05 17:25:37.348 TRACE nova parts = [self.recv(flags, copy=copy, track=track)]
2015-02-05 17:25:37.348 TRACE nova File "/usr/local/lib/python...

[zhangjialong (zhangjl)]

@Oleksii Zamiatin

You must start nova-conductor first, start the others next. Then the errors above would not be occured.

[Oleksii Zamiatin (ozamiatin)]

zhangjialong, thanks for your reply. Isn't it devstack issue to start services in apropriate order by default?

[Li Ma (nick-ma-z)]

I run the latest devstack, but everything goes well.

The output of the devstack:

Horizon is now available at http://10.88.88.2/
Keystone is serving at http://10.88.88.2:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: password
This is your host ip: 10.88.88.2

[Oleksii Zamiatin (ozamiatin)]

Could you please provide local.conf or write how do you configure it?

I replace rabbitmq with zeromq in stackrc and setup the variable ZEROMQ_MATCHMAKER="redis" is there anything else I forgot to do?

[Li Ma (nick-ma-z)]

Devstack installs oslo_messaging library from pip. The package from pip doesn't reflect the actual master branch of github. This is why you cannot make it work.

There's a critical bug fix that just merged in the upstream to make redis work [1]. I think it has not been released yet.

[1] https://github.com/openstack/oslo.messaging/commit/bd52d08efaa608244254a801c9737a0aed7434e0

[Li Ma (nick-ma-z)]

I modify the code by myself and finally devstack works.

[Oleksii Zamiatin (ozamiatin)]

Thanks for reply! Good to hear that it works. I'll check it and update the status.

[Li Ma (nick-ma-z)]

Well, I didn't notice that I changed the code before. Sorry for the late reply. You can try it later. If there's something wrong, please update this thread asap. Thanks.

[Oleksii Zamiatin (ozamiatin)]

Works fine on devstack.

local.conf:

[[local|localrc]]
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
RABBIT_PASSWORD=password

ZEROMQ_MATCHMAKER=redis
LIBS_FROM_GIT=oslo.messaging

    # core compute (glance / keystone / nova (+ nova-network))
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,n-sch,n-xvnc,n-cauth
    # cinder
ENABLED_SERVICES+=,c-sch,c-api,c-vol
    # heat
ENABLED_SERVICES+=,h-eng,h-api,h-api-cfn,h-api-cw
    # dashboard
ENABLED_SERVICES+=,horizon
    # additional services
ENABLED_SERVICES+=,zeromq,tempest,mysql