fq_codel qdisc compiled but not enabled on 22.10

[Dave Taht]

[ Impact ]
Starting with Ubuntu 22.10, the default qdisc regressed to pfifo_fast. This is due to a change we merged from Debian, removing systemd's /usr/lib/50-default.conf file. pfifo_fast does not prevent bufferbloat issues, severely degrading networking performance on Ubuntu systems.

[ Test Plan ]
On an Ubuntu VM, check that net.core.default_qdisc is set to 'fq_codel'. `tc qdisc show` should also confirm fq_codel is being used.

halves@noble-vm:~$ sysctl net.core.default_qdisc
net.core.default_qdisc = fq_codel

halves@noble-vm:~$ tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev enp5s0 root
qdisc fq_codel 0: dev enp5s0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64

[ Where Problems Could Occur ]
Users relying on the default pfifo_fast qdisc could see different networking behavior. Although fq_codel should be an across-the-board improvement over pfifo_fast, we could monitor network latency stats and throughput to make sure no weird regressions arise from the qdisc scheduler.

[ Other Info ]
Instead of adding a sysctl.conf file, Debian has decided to change their default kernel configs to use fq_codel. This has the disadvantage of being "hidden" from end users, instead of being a queryable setting in e.g. /etc/sysctl.d/, and all kernel documentation still refers to pfifo_fast as being the default.

Systemd's /usr/lib/50-default.conf file includes a lot of other settings that are either overruled elsewhere (e.g. from procps) or are entirely missing from our stable releases. Introducing a diff from Debian to ship this file only for the net.core.default_qdisc changes seems excessive, so having a dedicated place under procps where other similar toggles are already present seems like the more appropriate solution.

--
[ Original Description ]
fq_codel has been the default in ubuntu since #1436945

It is also now the default in debian. Somewhere between 22.04 and 22.10 though, it stopped being enabled by systemd? (it used to be enabled there via a sysctl), and thus 22.10 (I tried both ubuntu server and ubuntu studio) doesn't have it on. I rather *immediately* noticed this huge regression (I run my ethernet at 100Mbit, and thus immediately saw 400+ms latency) - and I hope it can be fixed in the next release?

it can also be made the default via compiling it into the kernel as a kernel default.

Aggh! death to fifos!

root@localhost:~# sysctl -a | grep qdisc
net.core.default_qdisc = pfifo_fast

disc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 5250 bytes 53 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

[Seth Arnold (seth-arnold)]

Very weird.

Oddly enough, there was a change in how the parameter was specified around systemd 248 -- but that change predates 22.04. The documentation suggests that it should just keep a service from failing in case the setting can't be set, so it feels like it shouldn't actually change anything here.

sarnold@wopr:/fst/trees/ubuntu/main/s/systemd $ grep -L -- 'net.core.default_qdisc = fq_codel' */sysctl.d/50-default.conf
systemd_204-5ubuntu20.31/sysctl.d/50-default.conf
systemd_204-5ubuntu20/sysctl.d/50-default.conf
sarnold@wopr:/fst/trees/ubuntu/main/s/systemd $ grep -- 'net.core.default_qdisc = fq_codel' */sysctl.d/50-default.conf
systemd_229-4ubuntu21.21/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu21.22/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu21.23/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu21.27/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu21.28/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu21.29/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu21.31/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_229-4ubuntu4/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.19/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.26/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.28/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.29/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.31/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.33/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.34/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.38/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.39/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.40/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.41/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.42/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.43/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.44/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.45/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.46/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.47/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.48/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.49/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.50/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.51/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.52/sysctl.d/50-default.conf:net.core.default_qdisc = fq_codel
systemd_237-3ubuntu10.53/sysctl.d/50-def...

[David Myers (demyers)]

I just got a Raspberry Pi 5 and after loading the Mantic image I noticed that the file /lib/sysctl.d/50-default.conf from systemd isn't present, causing the network interfaces to default to pfifo_fast.

This file is also missing from the build "ubuntu 24.04 LTS amd64 (daily) (20240210)" running as a VM under LXD.

I think this regression could be an unpleasant surprise for Jammy users when they migrate to Noble.

Thanks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Heitor Alves de Siqueira (halves)]

I've linked a merge proposal to fix this regression, introducing a new 10-bufferbloat.conf file under the procps package. The reasons for fixing this in procps vs e.g. our kernel config or even in systemd are on the bug description, but a quick summary:

- the systemd file that includes the default qdisc settings contains a lot of other tweaks that might not be wanted for Ubuntu or appropriate for SRU
- kernel configs are not easy to query or to verify by users, and might not map 1:1 to previous behavior (i.e. before the systemd regression took out the 50-defaults.conf file, we used the same sysctl to set the default qdisc)
- procps already includes lots of similar sysctl changes, so it seems like the natural place to have the bufferbloat setting

There's prior discussion on Launchpad about using fq_codel by default (bug 1773157, bug 1436945), and this has been the default for a long time now (at least back to Bionic). Once this has been fixed in the development release, I plan to backport a similar fix to Noble. Currently, users that upgrade to the latest LTS will probably be surprised by the change/regression in the default queueing discipline, as almost everything expects fq_codel nowadays due to bufferbloat.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dave Taht (dave-taht)]

Thank yall for addressing this!

On Wed, Aug 7, 2024, 12:50 PM Nick Rosbrook <email address hidden>
wrote:

> ** Changed in: procps (Ubuntu Oracular)
> Status: In Progress => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2003027
>
> Title:
> fq_codel qdisc compiled but not enabled on 22.10
>
> Status in Ubuntu:
> In Progress
> Status in procps package in Ubuntu:
> Fix Committed
> Status in The Noble Numbat:
> Confirmed
> Status in procps source package in Noble:
> Confirmed
> Status in The Oracular Oriole:
> In Progress
> Status in procps source package in Oracular:
> Fix Committed
>
> Bug description:
> [ Impact ]
> Starting with Ubuntu 22.10, the default qdisc regressed to pfifo_fast.
> This is due to a change we merged from Debian, removing systemd's
> /usr/lib/50-default.conf file. pfifo_fast does not prevent bufferbloat
> issues, severely degrading networking performance on Ubuntu systems.
>
> [ Test Plan ]
> On an Ubuntu VM, check that net.core.default_qdisc is set to 'fq_codel'.
> `tc qdisc show` should also confirm fq_codel is being used.
>
> halves@noble-vm:~$ sysctl net.core.default_qdisc
> net.core.default_qdisc = fq_codel
>
> halves@noble-vm:~$ tc qdisc show
> qdisc noqueue 0: dev lo root refcnt 2
> qdisc mq 0: dev enp5s0 root
> qdisc fq_codel 0: dev enp5s0 parent :1 limit 10240p flows 1024 quantum
> 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
>
> [ Where Problems Could Occur ]
> Users relying on the default pfifo_fast qdisc could see different
> networking behavior.
>
> [ Other Info ]
> Instead of adding a sysctl.conf file, Debian has decided to change their
> default kernel configs to use fq_codel. This has the disadvantage of being
> "hidden" from end users, instead of being a queryable setting in e.g.
> /etc/sysctl.d/, and all kernel documentation still refers to pfifo_fast as
> being the default.
>
> Systemd's /usr/lib/50-default.conf file includes a lot of other
> settings that are either overruled elsewhere (e.g. from procps) or are
> entirely missing from our stable releases. Introducing a diff from
> Debian to ship this file only for the net.core.default_qdisc changes
> seems excessive, so having a dedicated place under procps where other
> similar toggles are already present seems like the more appropriate
> solution.
>
> --
> [ Original Description ]
> fq_codel has been the default in ubuntu since #1436945
>
> It is also now the default in debian. Somewhere between 22.04 and
> 22.10 though, it stopped being enabled by systemd? (it used to be
> enabled there via a sysctl), and thus 22.10 (I tried both ubuntu
> server and ubuntu studio) doesn't have it on. I rather *immediately*
> noticed this huge regression (I run my ethernet at 100Mbit, and thus
> immediately saw 400+ms latency) - and I hope it can be fixed in the
> next release?
>
> it can also be made the default via compiling it into the kernel as a
> kernel default.
>
> Aggh! death to fifos!
>
> root@localhost:~# sysctl -a | grep qdisc
> net.core.defaul...

[Launchpad Janitor (janitor)]

This bug was fixed in the package procps - 2:4.0.4-4ubuntu4

---------------
procps (2:4.0.4-4ubuntu4) oracular; urgency=medium

  * d/sysctl.d/10-bufferbloat.conf: set default qdisc to fq_codel (LP: #2003027)

 -- Heitor Alves de Siqueira <email address hidden> Wed, 17 Jul 2024 14:59:18 +0000