[SRU] backport msgpack max_buffer_size patch to Focal

[Edward Hope-Morley]

[Impact]
Hosts running Openstack Neutron from Ussuri to Yoga release are impacted by
the fact that their use of oslo.privsep is thwarted by a default buffer size
on python-msgpack such that commands that return strings > 1MB in size cause
privsep to crash and agents to stop working. This patch backports a fix that
increased the buffer size to one that is more appropriate to neutron usage.

[Test Plan]

  * deploy Openstack (version corresponding to release of SRU) with l3ha
  * need at least one compute host and two neutron-gateway hosts
  * create a large number of routers (200 should do it) each with several networks attached and external gateway set
  * restart neutron-l3-agent
  * wait for restart to complete then do 'journalctl --unit neutron-l3-agent --grep ValueError'
  * if the error does not appear then the patch is working
  * note that it is necessary to create routers with enough networks and ports to trigger the > 1MB size limit for the netlink info message returned by privsep to trigger

[Where problems could occur]
No regressions are expected to occur as a result of this patch as it is
increasing the buffer size by a fairly small amount to allow neutron
function correctly in loaded environments.

--------------------------------------------------------------------------------

The main explanation for this backport can be found in https://bugs.launchpad.net/ubuntu/+source/python-oslo.privsep/+bug/1896734/comments/37 but I'm opening a new bug for the privsep backport since 1896734 was used to backport a neutron fix.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp2029952-jammy.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Edward Hope-Morley (hopem)]

for context it would appear that the main culprit for triggering this is when neutron calls to [1] on hosts that have large numbers of devices.

In ussuri this is from the following places:

$ egrep -r "\.get_devices(|_info)\(" neutron/{agent,plugins}
neutron/agent/windows/ip_lib.py: for device in self.get_devices():
neutron/agent/linux/dhcp.py: for d in ns_ip.get_devices():
neutron/agent/linux/ip_lib.py: return not self.get_devices()
neutron/agent/l3/namespaces.py: for d in ns_ip.get_devices():
neutron/agent/l3/dvr_fip_ns.py: devices = ip_wrapper.get_devices()
neutron/agent/l3/dvr_fip_ns.py: for d in ip_wrapper.get_devices():
neutron/agent/l3/dvr_edge_router.py: for d in ns_ip.get_devices():
neutron/agent/l3/dvr_snat_ns.py: for d in ns_ip.get_devices():
neutron/agent/l3/router_info.py: ip_devs = ip_wrapper.get_devices()
neutron/plugins/ml2/drivers/macvtap/agent/macvtap_neutron_agent.py: devices = ip_lib.IPWrapper().get_devices(True)
neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py: devices = self.ip.get_devices(True)

I have personally found that in a l3ha+dvr environment with > 200 routers, neutron hitting this limit when scanning the fip ns devices.

[1] https://github.com/openstack/neutron/blob/97429dc916c09641f63d69edb3759875a5798e78/neutron/agent/linux/ip_lib.py#L1439

[Corey Bryant (corey.bryant)]

Uploaded to focal and jammy unapproved queues as well as victoria, wallaby and xena staging PPAs

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into xena-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:xena-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-xena-needed to verification-xena-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-xena-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

> [Where problems could occur]

FTR, I was concerned about any OOM type conditions that might be introduced especially if controllable by a third party, but I saw this in the patch description:

> Also, since `msgpack` is only being used by the internal API, we're not worried about a third party coming in and overwhelming the system by deserializing calls.

However https://bugs.launchpad.net/ubuntu/+source/python-oslo.privsep/+bug/1896734/comments/37 talks of large number of ports. Is there any similar "large numbers" situation in which this change might introduce an OOM condition in practice, when it didn't before?

Accepting regardless - that's just something to look out for in consideration of testing. If you think it can't occur then it's fine :)

[Robie Basak (racb)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.privsep/2.7.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.privsep/2.1.1-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Unsubscribing ~ubuntu-sponsors as I believe there is nothing left to sponsor?

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into yoga-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:yoga-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-yoga-needed to verification-yoga-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-yoga-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Nicolas Bock (nicolasbock)]

I verified Focal / Ussuri

* Install python3-oslo.privsep-2.1.1-0ubuntu1
* Create several VMs
* Create hundreds of networks
* for (( i = 0; i < 20; i++ )); do for (( j = 1; j <= 6; j++ )); do id=$(uuidgen); openstack port create --network private X-${id}; openstack server add port jammy-133848-${j} X-${id}; done; done
* Observe error with journalctl --unit neutron-l3-agent.service --grep ValueError
* Install python3-oslo.privsep-2.1.1-0ubuntu2
* Attach more ports and verify that error is not present anymore.

[Edward Hope-Morley (hopem)]

jammy-proposed verified using [Test Plan] and the following output:

root@porygon:/home/ubuntu# apt-cache policy python3-oslo.privsep
python3-oslo.privsep:
  Installed: 2.7.0-0ubuntu2
  Candidate: 2.7.0-0ubuntu2
  Version table:
 *** 2.7.0-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.7.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@porygon:/home/ubuntu# ip netns exec fip-30a64e33-d202-4ca5-bf31-09298d83b366 ip -br a| wc -l
204
root@porygon:/home/ubuntu# ip netns| grep qrouter| wc -l
202
root@porygon:/home/ubuntu# journalctl --unit neutron-l3-agent --grep ValueError
-- No entries --

[Edward Hope-Morley (hopem)]

yoga-proposed verified using [Test Plan] and the following output:

root@porygon:/home/ubuntu# apt-cache policy python3-oslo.privsep
python3-oslo.privsep:
  Installed: 2.7.0-0ubuntu2~cloud0
  Candidate: 2.7.0-0ubuntu2~cloud0
  Version table:
 *** 2.7.0-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/yoga/main amd64 Packages
        100 /var/lib/dpkg/status
     2.1.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@porygon:/home/ubuntu# ip netns exec fip-12683dc3-f2ec-4617-b87a-88403b69bafa ip -br a| wc -l
204
root@porygon:/home/ubuntu# ip netns| grep qrouter| wc -l
202
root@porygon:/home/ubuntu# journalctl --unit neutron-l3-agent --grep ValueError
-- Logs begin at Fri 2023-09-08 11:46:08 UTC, end at Fri 2023-09-08 15:56:31 UTC. --
-- No entries --

[Edward Hope-Morley (hopem)]

xena-proposed verified using [Test Plan] and the following output:

root@porygon:/home/ubuntu# apt-cache policy python3-oslo.privsep
python3-oslo.privsep:
  Installed: 2.6.2-0ubuntu1~cloud1
  Candidate: 2.6.2-0ubuntu1~cloud1
  Version table:
 *** 2.6.2-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/xena/main amd64 Packages
     2.6.2-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/xena/main amd64 Packages
     2.1.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
     2.1.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@porygon:/home/ubuntu# ip netns exec fip-0360c16f-7566-4a3b-84e6-e5ede02f0c55 ip -br a| wc -l
204
root@porygon:/home/ubuntu# ip netns| grep qrouter| wc -l
202
root@porygon:/home/ubuntu# journalctl --unit neutron-l3-agent --grep ValueError
-- Logs begin at Fri 2023-09-08 16:13:43 UTC, end at Mon 2023-09-11 10:34:29 UTC. --
-- No entries --

[Edward Hope-Morley (hopem)]

wallaby-proposed verified using [Test Plan] and the following output:

root@porygon:/home/ubuntu# apt-cache policy python3-oslo.privsep
python3-oslo.privsep:
  Installed: 2.5.0-0ubuntu1~cloud1
  Candidate: 2.5.0-0ubuntu1~cloud1
  Version table:
 *** 2.5.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/wallaby/main amd64 Packages
        100 /var/lib/dpkg/status
     2.1.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@porygon:/home/ubuntu# ip netns exec fip-ec9c32a5-e6e2-4af3-827c-c48452fedc36 ip -br a| wc -l
204
root@porygon:/home/ubuntu# ip netns| grep qrouter| wc -l
202
root@porygon:/home/ubuntu# journalctl --unit neutron-l3-agent --grep ValueError
-- Logs begin at Mon 2023-09-11 10:49:05 UTC, end at Mon 2023-09-11 16:41:16 UTC. --
-- No entries --

[Edward Hope-Morley (hopem)]

victoria-proposed verified using [Test Plan] and the following output:

root@porygon:/home/ubuntu# apt-cache policy python3-oslo.privsep
python3-oslo.privsep:
  Installed: 2.4.0-0ubuntu1~cloud1
  Candidate: 2.4.0-0ubuntu1~cloud1
  Version table:
 *** 2.4.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/victoria/main amd64 Packages
        100 /var/lib/dpkg/status
     2.1.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@porygon:/home/ubuntu# ip netns exec fip-fce26f2b-6472-4491-96ed-6217c847055e ip -br a| wc -l
204
root@porygon:/home/ubuntu# ip netns| grep qrouter| wc -l
202
root@porygon:/home/ubuntu# journalctl --unit neutron-l3-agent --grep ValueError
-- Logs begin at Mon 2023-09-11 16:56:59 UTC, end at Tue 2023-09-12 09:39:18 UTC. --
-- No entries --

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.privsep - 2.1.1-0ubuntu2

---------------
python-oslo.privsep (2.1.1-0ubuntu2) focal; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

  [ Edward Hope-Morley ]
  * Backport max_buffer_size fix (LP: #2029952)
    - d/p/Bump-max_buffer_size-for-Deserializer.patch

 -- Edward Hope-Morley <email address hidden> Fri, 04 Aug 2023 14:56:14 +0100

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for python-oslo.privsep has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.privsep - 2.7.0-0ubuntu2

---------------
python-oslo.privsep (2.7.0-0ubuntu2) jammy; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/yoga branch.

  [ Edward Hope-Morley ]
  * Backport max_buffer_size fix (LP: #2029952)
    - d/p/Bump-max_buffer_size-for-Deserializer.patch

 -- Edward Hope-Morley <email address hidden> Fri, 04 Aug 2023 14:37:41 +0100

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.privsep has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.privsep - 2.7.0-0ubuntu2~cloud0
---------------

 python-oslo.privsep (2.7.0-0ubuntu2~cloud0) focal-yoga; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-oslo.privsep (2.7.0-0ubuntu2) jammy; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/yoga branch.
 .
   [ Edward Hope-Morley ]
   * Backport max_buffer_size fix (LP: #2029952)
     - d/p/Bump-max_buffer_size-for-Deserializer.patch

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.privsep has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.privsep - 2.6.2-0ubuntu1~cloud1
---------------

 python-oslo.privsep (2.6.2-0ubuntu1~cloud1) focal-xena; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/xena branch.
 .
   [ Edward Hope-Morley ]
   * Backport max_buffer_size fix (LP: #2029952)
     - d/p/Bump-max_buffer_size-for-Deserializer.patch

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.privsep has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.privsep - 2.5.0-0ubuntu1~cloud1
---------------

 python-oslo.privsep (2.5.0-0ubuntu1~cloud1) focal-wallaby; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/wallaby branch.
 .
   [ Edward Hope-Morley ]
   * Backport max_buffer_size fix (LP: #2029952)
     - d/p/Bump-max_buffer_size-for-Deserializer.patch

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.privsep has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.privsep - 2.4.0-0ubuntu1~cloud1
---------------

 python-oslo.privsep (2.4.0-0ubuntu1~cloud1) focal-victoria; urgency=medium
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/victoria branch.
 .
   [ Edward Hope-Morley ]
   * Backport max_buffer_size fix (LP: #2029952)
     - d/p/Bump-max_buffer_size-for-Deserializer.patch

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted python-oslo.privsep into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!