AMQP 2.0 prevents services from starting

[Jay Jahns]

We are running nova from the stable/mitaka branch (not using packages).

Our build process has flagged this. Previously, when we've built, we've used the requirements.txt file to pull in dependencies, and one of the dependencies pulls in AMQP.

Initial investigation leads us to believe that oslo.messaging pulls in AMQP as a dependency. It looks to be pulling in the latest version (2.0). Our previous successful builds show AMQP of 1.4.9, which is fully functional.

2.0 completely breaks, for example, the nova-compute service with the following trace:

2016-05-30 01:04:58.192 11335 CRITICAL nova [req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 'Connection' object has no attribute '_frame_writer'
2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/bin/nova-compute", line 10, in <module>
2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py", line 73, in main
2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=CONF.conductor.use_local)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 218, in create
2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 101, in init
2016-05-30 01:04:58.192 11335 ERROR nova self.conductor_api.wait_until_ready(context.get_admin_context())
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py", line 157, in wait_until_ready
2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", line 58, in ping
2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context, 'ping', arg=arg_p)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 157, in call
2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 91, in _send
2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 466, in send
2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 410, in _send
2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q': self._get_reply_q()})
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 382, in _get_reply_q
2016-05-30 01:04:58.192 11335 ERROR nova conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 373, in _get_connection
2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 396, in init
2016-05-30 01:04:58.192 11335 ERROR nova self.connection = connection_pool.create(purpose)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 110, in create
2016-05-30 01:04:58.192 11335 ERROR nova return self.connection_cls(self.conf, self.url, purpose)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 556, in init
2016-05-30 01:04:58.192 11335 ERROR nova self.ensure_connection()
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 654, in ensure_connection
2016-05-30 01:04:58.192 11335 ERROR nova self.ensure(method=lambda: self.connection.connection)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 759, in ensure
2016-05-30 01:04:58.192 11335 ERROR nova ret, channel = autoretry_method()
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2016-05-30 01:04:58.192 11335 ERROR nova return fun(*args, **kwargs)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 507, in call
2016-05-30 01:04:58.192 11335 ERROR nova self.revive(create_channel())
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel
2016-05-30 01:04:58.192 11335 ERROR nova chan = self.transport.create_channel(self.connection)
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 92, in create_channel
2016-05-30 01:04:58.192 11335 ERROR nova return connection.channel()
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/connection.py", line 380, in channel
2016-05-30 01:04:58.192 11335 ERROR nova channel.open()
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/channel.py", line 455, in open
2016-05-30 01:04:58.192 11335 ERROR nova spec.Channel.Open, 's', ('',), wait=spec.Channel.OpenOk,
2016-05-30 01:04:58.192 11335 ERROR nova File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 62, in send_method
2016-05-30 01:04:58.192 11335 ERROR nova conn._frame_writer.send((1, self.channel_id, sig, args, content))
2016-05-30 01:04:58.192 11335 ERROR nova AttributeError: 'Connection' object has no attribute '_frame_writer'
2016-05-30 01:04:58.192 11335 ERROR nova

Investigation also indicates this impacts other services. I've isolated builds into virtualenvs to prevent external environment from being a concern.

[Jay Jahns (jjahns)]

Further investigation shows this impacts ALL projects. Please investigate as soon as possible.

[Jay Jahns (jjahns)]

Added affected projects that we've tested so far. I have not specifically tested Glance in my sandbox. Overall, AMQP 2.0 is breaking everything. Our automation is working around the problem by uninstalling 2.0 and installing 1.4.9 manually.

Would like to hear feedback on this. Some folks are saying to use the upper-constraints file to install dependencies. This doesn't appear to be a solution to the common man.

[aeva black (tenbrae)]

Confirmed this also affects Ironic.

Searched and found this open bug in oslo.messaging, addressing the underlying issue:
https://bugs.launchpad.net/oslo.messaging/+bug/1586840

[Jay Jahns (jjahns)]

I added oslo.messaging to this. If it needs to be marked duplicate then that is fine, but considering nobody can build Openstack in this state, there needs to be some urgency on this. I tested a couple of workarounds, but the best one so far for us has been to uninstall amqp and reinstall the 1.4.9 version at the tail end of our automation.

The upper-constraints has 1.4.9 listed.

[Jay Jahns (jjahns)]

Would like to hear from folks - this is impacting a lot of projects, and people can't build from git branches...

IMO this is extremely critical and we need to get a response.

[Jay Jahns (jjahns)]

Master has the following:
https://github.com/openstack/oslo.messaging/blob/master/requirements.txt#L36

Mitaka has the following
https://github.com/openstack/oslo.messaging/blob/stable/mitaka/requirements.txt#L36

So someone fixed this, and the fix did not get put in the mitaka requirements? I really don't think *anyone* is running master in production so can someone amend the requirements file?

[Jay Jahns (jjahns)]

I'm posting that to the other bug as well.