linuxbridge agent crash after R ->S upgrade

Bug #1844822 reported by Rick Cano on 2019-09-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.privsep
Undecided
Unassigned

Bug Description

After upgrading neutron from Rocky to Stein (openstack-ansible deployment on ubuntu 16) I ran into an issue where the linuxbridge agent would crash on startup:

root@bctlpicrouter01:/var/log/neutron# /openstack/venvs/neutron-19.0.4.dev1/bin/neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini

Exception in thread privsep_reader:

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/oslo_privsep/comm.py", line 130, in _reader_main
    for msg in reader:
  File "/openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/six.py", line 564, in next
    return type(self).__next__(self)
  File "/openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/oslo_privsep/comm.py", line 77, in __next__
    return next(self.unpacker)
  File "msgpack/_unpacker.pyx", line 562, in msgpack._cmsgpack.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 493, in msgpack._cmsgpack.Unpacker._unpack
ValueError: 1870054 exceeds max_bin_len(1048576)

I was able to get around this problem by downgrading msgpack from 0.6.1 to 0.5.6

Brian Haley (brian-haley) wrote :

msgpack is actually included as a requirement of oslo.privsep, with the following requirement:

msgpack>=0.5.0 # Apache-2.0

From the changelog at https://github.com/msgpack/msgpack-python/blob/master/ChangeLog.rst it looks like max_bin_len was removed in 0.6.1, so perhaps there needs to be a limit set the version.

Will re-assign to oslo.privsep for further investigation.

affects: neutron → oslo.privsep
Rick Cano (canori01) wrote :

It days there that it's a "document only deprecation. Does that mean that it's still in the code and just discouraging ots use or that it has actually been removed?

Also, I should note that I upgraded another environment prior to this one (also an R->S upgrade), same version of msgpack but did not run into this issue there.

The environment that ran into the problem had far more neutron ports, tap interfaces, nsmespaces and linux bridges to wire up. So I wonder if that's why it ran into that limit while the other environment didn't.

Slawek Kaplonski (slaweq) wrote :

IHMO this might be that this error is side effect of some other issue. Maybe You hit this error while trying to handle some other, real issue? Can You maybe attach more neutron logs from this agent?

Rick Cano (canori01) wrote :

I'm attaching the logs. They're pretty large. I was experiencing some issues with the rootwrap filter. You helped me with that on irc, but I'm not sure if that was related to this because while implementing the rootwrap filter you suggested took care of the permission errors I was seeing, the linuxbridgeagent was still crashing after fixing that.

If you like, I could try bumping the version of msgpack back up on one of my three router nodes on and generate a new fresh set of logs as I'm sure the ones I'm attaching must be very convoluted

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers