msgpack version causing linuxbridgeagent to fail on ubuntu 18

[Rick Cano]

After upgrading neutron from Rocky to Stein and then again from Stein to Train (openstack-ansible deployment on ubuntu 18) I ran into an issue where the linuxbridge agent would crash on startup:

root@bctlpicrouter01:/var/log/neutron# /openstack/venvs/neutron-19.0.4.dev1/bin/neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini

Exception in thread privsep_reader:

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/oslo_privsep/comm.py", line 130, in _reader_main
    for msg in reader:
  File "/openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/six.py", line 564, in next
    return type(self).__next__(self)
  File "/openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/oslo_privsep/comm.py", line 77, in __next__
    return next(self.unpacker)
  File "msgpack/_unpacker.pyx", line 562, in msgpack._cmsgpack.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 493, in msgpack._cmsgpack.Unpacker._unpack
ValueError: 1870054 exceeds max_bin_len(1048576)

I was able to get around this problem by downgrading msgpack from 0.6.1 to 0.5.6

This only happens on larger deployents. I guess if the deployment is small enough, you don't exceed man_bin_len. For some reason,msgpack 0.5.6 either has a higher limit or doesn't enforce it.

[Dmitriy Rabotyagov (noonedeadpunk)]

It is related to https://bugs.launchpad.net/oslo.privsep/+bug/1844822

At the moment msgpack 1.0 has also fixed the issue by bumping the limit but 1.0 does not support py2.7 anymore.

So I think that it's actually oslo.privsep general issue and worth fixing on their side

[Rick Cano (canori01)]

We don't need python 2.7 as of train though, right? Does that mean we could bump the version for msgpack in the venv?

[Dmitriy Rabotyagov (noonedeadpunk)]

Well, on train I think we still run CentOS 7 on py2, but Ubuntu is using py3.6. However you was saying about stein usage.

For this case I can only suggest adding msgpack==1.0 (or appropriate version) in https://opendev.org/openstack/openstack-ansible/src/branch/master/global-requirement-pins.txt
This file is used while building venvs and has prescedence.

We can't fix this in code since we shoould follow upper-constraints, which force us to use vulnerable version.