python3-msgpack package broken due to outdated cython
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Invalid
|
Undecided
|
Unassigned | ||
Ussuri |
Fix Released
|
Medium
|
Unassigned | ||
neutron |
New
|
Undecided
|
Unassigned |
Bug Description
After a successful upgrade of the control-plance from Train -> Ussuri on Ubuntu Bionic, we upgraded a first compute / network node and immediately ran into issues with Neutron:
We noticed that Neutron is extremely slow in setting up and wiring the network ports, so slow it would never finish and throw all sorts of errors (RabbitMQ connection timeouts, full sync required, ...)
We were now able to reproduce the error on our Ussuri DEV cloud as well:
1) First we used strace -ffff -p $PID_OF_
One could actually read line by line the read calls to the fd of the socket.
2) We then (after adding lots of log lines and other intensive manual debugging) used py-spy (https:/
3) Since the issue was not observed in TRAIN we compared the msgpack version used and noticed that TRAIN was using version 0.5.6 while Ussuri upgraded this dependency to 0.6.2.
4) We then downgraded to version 0.5.6 of msgpack (ignoring the actual dependencies)
--- cut ---
apt policy python3-msgpack
python3-msgpack:
Installed: 0.6.2-1~cloud0
Candidate: 0.6.2-1~cloud0
Version table:
*** 0.6.2-1~cloud0 500
500 http://
0.5.6-1 500
500 http://
100 /var/lib/
--- cut ---
vs.
--- cut ---
apt policy python3-msgpack
python3-msgpack:
Installed: 0.5.6-1
Candidate: 0.6.2-1~cloud0
Version table:
0.6.2-1~cloud0 500
500 http://
*** 0.5.6-1 500
500 http://
100 /var/lib/
--- cut ---
and et voila: The Neutron-
I could not yet spot which commit of msgpack changes (https:/
There are "similar" issues with
* https:/
* https:/
both related to msgpack or the size of messages exchanged.
affects: | ubuntu → neutron |
summary: |
- linuxbridge agent broken due to msgpack upgrade 0.6.2 for Ussuri on + linuxbridge agent broken due to msgpack upgrade to 0.6.2 for Ussuri on Bionic |
summary: |
- linuxbridge agent broken due to msgpack upgrade to 0.6.2 for Ussuri on - Bionic + msgpack upgrade to 0.6.2 for Ussuri on Bionic breaks linuxbridge agent |
summary: |
- msgpack upgrade to 0.6.2 for Ussuri on Bionic breaks linuxbridge agent + msgpack upgrade to 0.6.2 breaks linuxbridge agent |
description: | updated |
affects: | neutron (Ubuntu) → cloud-archive |
Changed in cloud-archive: | |
status: | New → Confirmed |
status: | Confirmed → Invalid |
no longer affects: | python-msgpack (Ubuntu) |
no longer affects: | python-oslo.privsep (Ubuntu) |
I dug a little deeper into the issue ... apparently the python3-msgpack package provided by Ubuntu Cloud Archive does not contain the cmsg extension - that's why msgpack is using its pure python, but slow, fallback.
python3- msgpack= 0.5.6 from Ubuntu Bionic contains:
--- cut -- python3/ dist-packages python3/ dist-packages/ msgpack python3/ dist-packages/ msgpack/ __init_ _.py python3/ dist-packages/ msgpack/ _packer. cpython- 36m-x86_ 64-linux- gnu.so python3/ dist-packages/ msgpack/ _unpacker. cpython- 36m-x86_ 64-linux- gnu.so python3/ dist-packages/ msgpack/ _version. py python3/ dist-packages/ msgpack/ exceptions. py python3/ dist-packages/ msgpack/ fallback. py python3/ dist-packages/ msgpack- 0.5.6.egg- info python3/ dist-packages/ msgpack- 0.5.6.egg- info/PKG- INFO python3/ dist-packages/ msgpack- 0.5.6.egg- info/dependency _links. txt python3/ dist-packages/ msgpack- 0.5.6.egg- info/top_ level.txt doc/python3- msgpack doc/python3- msgpack/ README. rst.gz doc/python3- msgpack/ changelog. Debian. gz doc/python3- msgpack/ copyright python3/ dist python3/ dist/python3- msgpack
# dpkg -L python3-msgpack
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/share
/usr/share/doc
/usr/share/
/usr/share/
/usr/share/
/usr/share/
/usr/share/python3
/usr/share/
/usr/share/
--- cut
while python3- msgpack= 0.6.2 from Cloud Archive lists ...
--- cut --- python3/ dist-packages python3/ dist-packages/ msgpack python3/ dist-packages/ msgpack/ __init_ _.py python3/ dist-packages/ msgpack/ _version. py python3/ dist-packages/ msgpack/ exceptions. py python3/ dist-packages/ msgpack/ fallback. py python3/ dist-packages/ msgpack- 0.6.2.egg- info python3/ dist-packages/ msgpack- 0.6.2.egg- info/PKG- INFO python3/ dist-packages/ msgpack- 0.6.2.egg- info/dependency _links. txt python3/ dist-packages/ msgpack- 0.6.2.egg- info/top_ level.txt doc/python3- msgpack doc/python3- msgpack/ README. rst.gz doc/python3- msgpack/ changelog. Debian. gz doc/python3- msgpack/ copyright python3/ dist python3/ dist/python3- msgpack
# dpkg -L python3-msgpack
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/share
/usr/share/doc
/usr/share/
/usr/share/
/usr/share/
/usr/share/
/usr/share/python3
/usr/share/
/usr/share/
--- cut ---
this is likely due to a missing cython on the build system which is gracefully "ignored" by the msgpack setup.py, see: https:/ /github. com/msgpack/ msgpack- python/ blob/38dba9634e 4efa7886a777b9e 7c739dc148da457 /setup. py#L54
TL;DR: python3-msgpack provided for Ussuri on Ubuntu-Bionic lacks cmsg due to a potentially missing cython dependency on build system.