Mantic (23.10) minimal images increase in memory consumption, port usage and processes running

[Philip Roche]

The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/
are undergoing some big changes prior to 23.10 release in October.

This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected.

This bug is to track the unexpected changes and discuss/resolve these.

The changes that have been made to mantic minimal:

* Move to the linux-generic kernel from the linux-kvm kernel
  * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745
* Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal
* No longer installing Recommends packages
  * This is during image build only and will not affect any subsequent package installs
* No initramfs fallback for boot - only initramfsless boot

The latest mantic minimal images are available @ http://cloud-images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds.

A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/

We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system.

To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal-LP2032933/

* 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64
  * Before kernel change and before seed change
* 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64
  * After kernel change and before seed change
* 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64
  * After kernel change and after seed change

[Steve Langasek (vorlon)]

> higher number of process running on a idle system.

Are the details on this available?

[Simon Poirier (simpoir)]

The big difference I see (and didn't expect) is wpa_supplicant running.

[Philip Roche (philroche)]

@vorlon I am gathering all data now. You can see visually the increase @ https://cloud.kpi.canonical.com/d/ilYWcb-Vk/ubuntu-server-daily-metrics?orgId=1&var-selectedrelease=mantic (internal to Canonical only) server team metrics dashboard.

[Philip Roche (philroche)]

metrics data now uploaded to https://people.canonical.com/~philroche/20230824-manticl-minimal-LP2032933/server-metrics/. This was gathered using server teams' tooling @ https://github.com/canonical/server-test-scripts/blob/main/metric-server-simple/metric-server-simple.sh using custom imported lxc VM images from https://people.canonical.com/~philroche/20230824-manticl-minimal-LP2032933/

For processes and ports etc. on idle system, see the `*-early.txt` metrics.

[Christian Ehrhardt  (paelzer)]

Related to this (not taking all that missed memory, but part of it) there is also a set of changes to the number of processes, most likely due to the kernel providing more support. Here a breakdown of those:

+4: a bit more rcu action due to the different/new kernel rcu_preempt, rcu_tasks_rude_kthread, rcu_tasks_trace_kthread and one more kworker for rcu

+1: a bit more FS action with writeback and ext4-rsv-conversion kworker threads

+3 way more event handlers 2->5 events_unbound kworkers

+1 events_power_efficient kworker

-3 but -29 on general processes, mostly kernel feature related tasks
+acpi_thermal_pm
+agetty
agetty
ata_sff
blkcg_punt_bio
+charger_manager
cpuhp/0
+cryptd
dbus-daemon
+devfreq_wq
+ecryptfs-kthread
+edac-poller
ext4-rsv-conver
hwrng
+idle_inject/0
inet_frag_wq
ipv6_addrconf
-jbd2/sda1-8

+irq/24-aerdrv
+irq/25-aerdrv
+irq/26-aerdrv
+irq/27-aerdrv
+irq/28-aerdrv
+irq/29-aerdrv
+irq/30-aerdrv
+irq/31-aerdrv
+irq/32-aerdrv
+jbd2/sda1-8
kauditd
kblockd
+kcompactd0
kdevtmpfs
+khugepaged
+khungtaskd
+kintegrityd
+ksmd
ksoftirqd/0
kstrp
kswapd0
kthreadd
kthrotld
lxd-agent
+md
migration/0
mld
mm_percpu_wq
netns
oom_reaper
ps
-nfit
-polkitd
slub_flushwq
+tpm_dev_wq
+(udev-worker)
+watchdogd
writeback
+xenbus_probe
+zswap-shrink

[Christian Ehrhardt  (paelzer)]

I've stated this elsewhere, but let me add it here as well.
The easiest and quick way to see the change yourself is:

$ lxc launch ubuntu-minimal-daily:c0e439b7fc83 m-old --ephemeral --vm
$ lxc exec m-old -- /usr/bin/cat /proc/meminfo
MemTotal: 979256 kB
MemFree: 724524 kB
MemAvailable: 882544 kB
...

$ lxc launch ubuntu-minimal-daily:m m-new --ephemeral --vm
$ lxc exec m-new -- /usr/bin/cat /proc/meminfo | head
MemTotal: 955360 kB
MemFree: 662656 kB
MemAvailable: 707088 kB
...

Most of the other results we see in the metrics are follow on issues, like systems OOM crashing, restarting and going into recovery.

P.S. I'm currently working on changing the memory, so that I can avoid the crashes and compare jammy, lunar, mantic without those. But that is only me trying to get useful metrics again, not fixing or helping with the memory increase seen here.
Sadly the test environment is in a really bad state and thereby stalling this over the weekend (Paride is on that).

[Christian Ehrhardt  (paelzer)]

Checking kernels:

1. This already uses linux-image-virtual, it is not the even bigger linux-image-generic.
2. The change of the image build sadly combined it all
  a) new image build/seeding by CPC
  b) different kernel type -kvm -> -virtual that uses generic
  c) switch of kernel versions

We did not yet think about 2c here afaics.

This is not just -kvm -> -generic.
But instead 6.2.0-1003-kvm -> 6.3.0-7-generic

6.3.0-7-kvm does not exist (not built anymore), so to check which of the changes broke this we might need to look at.

The closest comparison I'd find is
- https://launchpad.net/ubuntu/+source/linux/6.2.0-21.21
- https://launchpad.net/ubuntu/+source/linux-signed/6.2.0-21.21

Installing that on my system

The new system with the high memory consumption has:
  linux-image-6.3.0-7-generic 6.3.0-7.7+1
  linux-modules-6.3.0-7-generic 6.3.0-7.7

I'm comparing for:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/25980736/+files/linux-modules-6.2.0-21-generic_6.2.0-21.21_amd64.deb
https://launchpad.net/~canonical-signing/+archive/ubuntu/primary-2022v1/+build/25984194/+files/linux-image-6.2.0-21-generic_6.2.0-21.21_amd64.deb

root@m-old:~# cat /proc/meminfo
MemTotal: 955692 kB
MemFree: 630128 kB
MemAvailable: 653536 kB

So that lets us assume, that the move 6.2->6.3 was not what broke it (actually 6.3 is even a little bit better).
The cause really seems to be more the move -kvm -> -generic.

I'm sure some change was expected, but so much?
Adding Linux bug tasks for them to chime in.

[Christian Ehrhardt  (paelzer)]

One thing that came up when discussing with SMB as an obvious "uses mem more in a kernel" are the structures needed per POSSIBLE cpu.

Comparing the system that I downgraded I found from [1]:

With the -kvm kernel before:
kernel: setup_percpu: NR_CPUS:64 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
kernel: rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.

With the -generic kernel:
kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
kernel: rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.

So let us try the new 6.3 generic kernel, but with kernel parameter
  maxcpus=1 possible_cpus=1 nr_cpus=1
^^ this might be too hard, but ok for testing

That gives me:
kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

But sadly, the memory does not look much better:
root@m-new:~# cat /proc/meminfo
MemTotal: 956176 kB
MemFree: 640960 kB
MemAvailable: 664816 kB

Was a good idea, but sadly that wasn't it.
And the initial NR_CPUS is a compile time parameter which we can not tune down :-/

[1]: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/setup_percpu.c#L122

[Christian Ehrhardt  (paelzer)]

IMHO: If it turns out unfixable, this at least needs to be a release notes entry

[Simon Déziel (sdeziel)]

In the above tests from Christian, it's interesting to note the MemTotal shrink by ~22MiB. Is this due to the higher NR_CPUS alone?

[Philip Roche (philroche)]

@paelzer

> The change of the image build sadly combined it all

See the description noting https://people.canonical.com/~philroche/20230824-manticl-minimal-LP2032933/ which should help in determining where the changes were introduced as I have provided three images across the various stages of changes - no change -> new kernel -> new kernel + new seed

For example https://pastebin.ubuntu.com/p/sJ5wGk4G7h/ show the process diff between previous images and images with new kernel only

[Philip Roche (philroche)]

The diff in process count from kernel change image -> kernel change + seed change image is actually a reduction in processes - see diff @ https://pastebin.ubuntu.com/p/PXtQM9gB2K/

[Seth Arnold (seth-arnold)]

> +ksmd

I'm concerned about this change. Historically, the page-merging code has allowed cross-VM snooping, including even recovery of GnuPG private keys: https://eprint.iacr.org/2013/448.pdf

Unless something has changed to mitigate the cross-domain privacy leaks in ksmd, it ought to be opt-in for administrators to select if all their VMs are in the same security domain.

Thanks

[Dimitri John Ledkov (xnox)]

Separately, please do note that switch from kvm kernel to generic is resolving the hundreds of bug reports of "works with generic / every other cloud, doesn't work with kvm". Also note that probably the baseline comparison shouldn't be the kvm kernel => as we know kvm kernel has never provided complete Ubuntu experience the way cloud kernels do (aws,azure,gcp,oracle) nor the generic kernel (most baremetal installs).

[Dimitri John Ledkov (xnox)]

Additionally if something is built into the kernel that is optional at runtime, and resource intensive, we can make it modularization. But not sure that will win us anything if it gets autoloaded by default. Thus maybe some of the things listed should also be blacklisted to prevent auto-loading on boot. Or even filtered / excluded from the minimal image (i.e. demotion to modules-extra, or dpkg filter-package skipped for minimal images).

[Dimitri John Ledkov (xnox)]

Is lxd / zfs now installed by default in the minimal images? It seems like some zfs things are loaded by default.

Also counting processes alone, may or may not increase memory usage. Example: having N agetty, doesn't actually have memory cost of each individual agetty, does it?

[Philip Roche (philroche)]

> Is lxd / zfs now installed by default in the minimal images? It seems like some zfs things are loaded by default.

The packages in the new minimal image do not includes lxd; they do include the lxd-installer shim package though. I see no zfs package by default. See manifest @ http://cloud-images.ubuntu.com/minimal/daily/mantic/current/mantic-minimal-cloudimg-amd64.manifest

[Philip Roche (philroche)]

>> Is lxd / zfs now installed by default in the minimal images? It seems like some zfs things are loaded by default.

> The packages in the new minimal image do not includes lxd; they do include the lxd-installer shim package though. I see no zfs package by default. See manifest @ http://cloud-images.ubuntu.com/minimal/daily/mantic/current/mantic-minimal-cloudimg-amd64.manifest

I do see it in the kernel change only image, 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64, which I suspect what you were seeing. This is likely because this image was installed with `recommends` of all package installs including the kernel. Later image, with seed changes etc., do not install `recommends`.

[Philip Roche (philroche)]

I have uploaded further data now to https://people.canonical.com/~philroche/20230824-manticl-minimal-LP2032933/server-metrics/ with kernelmodules, kernelconfig, services, timers etc. for each of the three images being inspected. This additional data was gathered with a modified fork of the `server-test-scripts` repo @ https://github.com/philroche/server-test-scripts/blob/feature/local-lxc-image-execution-additional-data-gathering/metric-server-simple/metric-server-simple.sh.

It seems that most of the mem and process increase is attributed to the kernel change and we know that this was a conscious decision with the following reported bugs supporting that decision.

* https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2006488
* https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931841
* https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1685291

Given the above I feel what we can work on is whether any of the process/modules introduced by the switch to the generic kernel should be omitted for the minimal images.

The best, easiest source of this information is the data gathered from the latest image with both the generic kernel and the switch to the new minimal seed - https://people.canonical.com/~philroche/20230824-manticl-minimal-LP2032933/server-metrics/20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64-data-f93870221eb8/

@seth-arnold You highlighted `ksmd`. Are there any others that concern you.

@paelzer Are you happy to adjust your regression testing/metrics gathering to increase the memory required knowing that it was a conscious decision to switch kernel and incur the performance hit for the benefit of using a kernel with more support and less reported bugs?

[Christian Ehrhardt  (paelzer)]

> @paelzer Are you happy to adjust your regression testing/metrics gathering to increase the
> memory required knowing that it was a conscious decision to switch kernel and incur the
> performance hit for the benefit of using a kernel with more support and less reported bugs?

I am.
In fact I already have.

To be clear, I knew that the change from -kvm to -virtual was a conscious decision (I do not even know how long back I heard xnox talking about it, many years for sure) fixing the many issues xnox mentioned. I just never heard anyone saying or suggesting - and personally didn't expect it to be - that very intense on low-memory / density scenarios.

I'd be happy if everyone here could continue the thought of checking if any further package/module could be omitted (or at least not auto-loaded) for the minimal images. Which should tune us back down a bit, even if that is not all the way to where we were before.

[Thomas Bechtold (toabctl)]

@Seth,

ksm is disabled by default so it's still opt-in:

# cat /sys/kernel/mm/ksm/run
0

it only gets activated when you install ksmtuned (which is not installed by default). So I think that's fine.

[Christian Ehrhardt  (paelzer)]

> it only gets activated when you install ksmtuned (which is not installed by default).

No, installing qemu-system-... will also enable it.
So Seth gladly filed bug 2033565 to discuss and change this now or at least towards 24.04.

[Thomas Bechtold (toabctl)]

Ah. I see. KSM_ENABLED=AUTO from /etc/default/qemu-kvm is used through /usr/share/qemu/init/qemu-kvm-init via the qemu-kvm.service systemd service. thanks for the hint!

[Philip Roche (philroche)]

@paelzer given the above findings and discussion, I would like to mark this as Invalid for cloud-images project and continue the conversation in the context of kernel only. +1 / -1 ?

[Christian Ehrhardt  (paelzer)]

@phil
As I said above for me it mostly was "I'm sure some change was expected, but so much?".

It has various benefits as gladly outlined by Dimitri, fixing many issues, but coming at a price tag.

Seeing how big the price tag is for small size, high density cases I consider it potentially too much for some. As I said in my report "This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected." so a way forward IMHO would be:

1. We leave mantic as is
2. We certainly need a release notes entry that explains the pros/cons that led to the decision
3. kernel can continue to optimize this case (see all the discussions/suggestions above already)
4. towards 24.04
 4a) if this is received well, keep it
 4b) if our decision was good, but not accounting for too many use-cases that we can't fix otherwise reconsider having a reduced kernel in 24.04

That would mostly an ask to the kernel team (but needs CPC to ensure it is correct e.g. in making it clear which images change due to that and which won't as they were already based on -generic).

Phil/Xnox - what do you think about that approach?

[Philip Roche (philroche)]

@paelzer agreed. Good plan.

[Philip Roche (philroche)]

There is a related bug @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036968 which might have affected boot speed.

[Philip Roche (philroche)]

https://bugs.launchpad.net/cloud-images/+bug/2038894 is a related bug to track specifically the introduction of listening port 5353