Kernel 6.1 bumped the disk consumption on default images by 15%

[Christian Ehrhardt ]

[Impact]

After enabling Rust support in the kernel we noticed an increased size in linux-headers, because all the Rust libraries required to build out-of-tree kernel modules are now included in linux-headers directly.

The increased size goes from ~30MB to ~100MB.

This extra size is not worth it, considering the small amount of users that are going to experiment Rust in the kernel at the moment.

[Test case]

Install linux-headers and check the size of /usr/src/linux-headers-`uname -r`.

[Fix]

Some work has already done to reduce the size of linux-headers by dropping some binary artifacts (*.o and *.cmd) that are not required to build out-of-tree kernel modules.

Ideally it'd be nicer to provide a separate package (called for example linux-lib-rust-`uname -r`) to ship the Rust-specific dependencies (libraries / headers).

In this way linux-headers would be restored back to the original size, those that want to experiment Rust in the kernel can simply install linux-lib-rust and we can prevent wasting space in the cloud images.

[Regression potential]

Introducing a new package in the kernel may break something in our kernel promotion workflow, so with this change applied we need to pay a little more attention when the kernel is going to be promoted to release/updates.

[Original bug report]

Hi,
in the regular monitoring of metrics there was a bump in disk size last week.
The default image once installed consumes now 994 (formerly 859) megabytes.

Looking at the consumers there was much noise, but the biggest change was

linux-headers-5.19.0-1008-kvm 25197 -> linux-headers-6.2.0-1002-kvm 108072
linux-modules-5.19.0-1008-kvm 76090 -> linux-modules-6.2.0-1002-kvm 97221

I haven't looked deeper yet. Is that an accident and will be fixed soon, is that a change we can do nothing about, anything in between?

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2015867

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Christian Ehrhardt  (paelzer)]

This is ran on daily images, the kernel versions are already listed above.
Hence no need to ran apport-collect for this issue.
Setting to "Confirmed"

[Paolo Pisati (p-pisati)]

These are the size of linux-kvm debs at the time of this writing[1]:

    linux-buildinfo-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb (397.9 KiB)
    linux-headers-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb (15.9 MiB)
    linux-image-unsigned-6.2.0-1003-kvm-dbgsym_6.2.0-1003.3_amd64.ddeb (203.2 MiB)
    linux-image-unsigned-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb (8.5 MiB)
    linux-kvm-headers-6.2.0-1003_6.2.0-1003.3_all.deb (12.5 MiB)
    linux-kvm-tools-6.2.0-1003_6.2.0-1003.3_amd64.deb (7.2 MiB)
    linux-kvm_6.2.0-1003.3.diff.gz (5.1 MiB)
    linux-kvm_6.2.0-1003.3.dsc (4.6 KiB)
    linux-kvm_6.2.0.orig.tar.gz (208.4 MiB)
    linux-modules-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb (17.0 MiB)
    linux-tools-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb (1.7 KiB)

that looks different from the above numbers.

1: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+packages?field.name_filter=&field.status_filter=published&field.series_filter=lunar

[Christian Ehrhardt  (paelzer)]

This isn't about the deb size, but about the size as extracted on the system.
Especially headers will most likely compress really well into the deb.

Comparing the latest 5.19 with the recent 6.2 [1][2] that got through propose migration we can see the increase just fine. debsizes as downloaded and sizes as extracted on disk (just dpkg -x, you do not need a system to install):
[I'm sure LP display will break this, looks reasonable in monospace) - [5] is a public gspread of the same content.

5.19-deb 6.2-deb increase 5.19-extract 6.2-extract increase name
163560 407486 149.14% 805888 1188856 47.52% linux-buildinfo
2909154 16637576 471.90% 25134419 110300530 338.84% linux-headers
7342272 8888512 21.06% 7341452 8890656 21.10% linux-image-unsigned
197923488 213022492 7.63% 1190567058 1294452144 8.73% linux-image-unsigned-dbgsym
12582164 13126606 4.33% 81571446 84147433 3.16% linux-kvm-headers
7890812 7575524 -4.00% 27081556 26699145 -1.41% linux-kvm-tools
15549556 17799166 14.47% 78035966 89312382 14.45% linux-modules
1756 1758 0.11% 34409 34401 -0.02% linux-tools

Further [3][4] show that this isn't specific to the -kvm kernel. It is in the generic kernel as well. New code might have more modules, so while it is quite a lot and we need to ask ourself if anything could be done it might be a necessary step. But the sources, they have grown so much and are default installed. The bump to default image size is quite much IMHO - enough to discuss about it here at least.

[1]: https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/devel/+build/24551183
[2]: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/25964206
[3]: https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/devel/+build/24547812
[4]: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/25964139
[5]: https://docs.google.com/spreadsheets/d/1hJ3cQpwDJ-NCcGTZfZD2zX7tpq8XRhIifYtnZL29sVU

[Christian Ehrhardt  (paelzer)]

Just picking on linux-headers for now it is more or less what I expected.

It went 26 -> 107 mb and I see new directories for rust of:
81M linux-headers-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb.extract/usr/src/linux-headers-6.2.0-1003-kvm/rust
228K linux-headers-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb.extract/usr/src/linux-headers-6.2.0-1003-kvm/include/generated

I know linux-headers are meant to be the full source, but this increases the size of a default installed (and therefore also often downloaded) package by x4.
You see that even the compressed size increased, actually even more than extracted.

I'm not sure we can, but could we think about separating that and not installing by default.
I'm afraid that might break things but it juts feels rather "expensive" in terms of disk space, image size, regular download ...

OTOH, looking at it maybe we can do something...
I know rust is using static linking and it looks like here also a lot pre-generated content.

/tmp/kernel-sizes/6.2/linux-headers-6.2.0-1003-kvm_6.2.0-1003.3_amd64.deb.extract/usr/src/linux-headers-6.2.0-1003-kvm/rust:

$ ll -S | head
total 81876
-rw-r--r-- 1 paelzer paelzer 54727271 Apr 6 09:50 libcore.rmeta
-rw-r--r-- 1 paelzer paelzer 18678432 Apr 6 09:50 libmacros.so
-rw-r--r-- 1 paelzer paelzer 5380859 Apr 6 09:50 libbindings.rmeta
-rw-r--r-- 1 paelzer paelzer 2835696 Apr 6 09:50 core.o
-rw-r--r-- 1 paelzer paelzer 904778 Apr 6 09:50 liballoc.rmeta
-rw-r--r-- 1 paelzer paelzer 589288 Apr 6 09:50 bindings.o
-rw-r--r-- 1 paelzer paelzer 186221 Apr 6 09:50 libkernel.rmeta
-rw-r--r-- 1 paelzer paelzer 106528 Apr 6 09:50 kernel.o
-rw-r--r-- 1 paelzer paelzer 99356 Apr 6 09:50 .bindings.o.cmd

Since these are meant to be sources, could we provide them in a way to be regeneratable but not of the binary size and format in linux-headers?

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws/6.2.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/6.2.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm/6.2.0-1002.2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp/6.2.0-1006.6 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-kvm/6.2.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oracle/6.2.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-hwe-6.2/6.2.0-23.23~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-6.2/6.2.0-1003.3~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Andrei Gherzan (agherzan)]

I have checked this with a lunar derivative (aws), and I can confirm that the size decreased as expected:

from
112M 1003/usr/
to
26M 1005/usr/

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.2.0-23.23

---------------
linux (6.2.0-23.23) lunar; urgency=medium

  * lunar/linux: 6.2.0-23.23 -proposed tracker (LP: #2019845)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
    - debian/dkms-versions -- update from kernel-versions (main/2023.05.15)

  * Fix flicker display problem on some panels which support PSR2 (LP: #2002968)
    - drm/i915/psr: Add continuous full frame bit together with single

  * Kernel 6.1 bumped the disk consumption on default images by 15%
    (LP: #2015867)
    - [Packaging] introduce a separate linux-lib-rust package

  * Update I915 PSR calculation on Linux 6.2 (LP: #2018655)
    - drm/i915: Fix fast wake AUX sync len
    - drm/i915: Explain the magic numbers for AUX SYNC/precharge length

  * Computer with Intel Atom CPU will not boot with Kernel 6.2.0-20
    (LP: #2017444)
    - [Config]: Disable CONFIG_INTEL_ATOMISP

  * udev fails to make prctl() syscall with apparmor=0 (as used by maas by
    default) (LP: #2016908)
    - SAUCE: (no-up) Stacking v38: Fix prctl() syscall with apparmor=0

  * CVE-2023-32233
    - netfilter: nf_tables: deactivate anonymous set from preparation phase

  * CVE-2023-2612
    - SAUCE: shiftfs: prevent lock unbalance in shiftfs_create_object()

  * CVE-2023-31436
    - net: sched: sch_qfq: prevent slab-out-of-bounds in qfq_activate_agg

  * CVE-2023-1380
    - wifi: brcmfmac: slab-out-of-bounds read in brcmf_get_assoc_ies()

  * 5.19 not reporting cgroups v1 blkio.throttle.io_serviced (LP: #2016186)
    - SAUCE: blk-throttle: Fix io statistics for cgroup v1

  * LSM stacking and AppArmor for 6.2: additional fixes (LP: #2017903)
    - SAUCE: (no-up) apparmor: fix policy_compat perms remap for file dfa
    - SAUCE: (no-up) apparmor: fix profile verification and enable it
    - SAUCE: (no-up) apparmor: fix: add missing failure check in
      compute_xmatch_perms
    - SAUCE: (no-up) apparmor: fix: kzalloc perms tables for shared dfas

  * Lunar update: v6.2.12 upstream stable release (LP: #2017219)
    - Revert "pinctrl: amd: Disable and mask interrupts on resume"
    - drm/amd/display: Pass the right info to drm_dp_remove_payload
    - drm/i915: Workaround ICL CSC_MODE sticky arming
    - ALSA: emu10k1: fix capture interrupt handler unlinking
    - ALSA: hda/sigmatel: add pin overrides for Intel DP45SG motherboard
    - ALSA: i2c/cs8427: fix iec958 mixer control deactivation
    - ALSA: hda: patch_realtek: add quirk for Asus N7601ZM
    - ALSA: hda/realtek: Add quirks for Lenovo Z13/Z16 Gen2
    - ALSA: firewire-tascam: add missing unwind goto in
      snd_tscm_stream_start_duplex()
    - ALSA: emu10k1: don't create old pass-through playback device on Audigy
    - ALSA: hda/sigmatel: fix S/PDIF out on Intel D*45* motherboards
    - ALSA: hda/hdmi: disable KAE for Intel DG2
    - Bluetooth: L2CAP: Fix use-after-free in l2cap_disconnect_{req,rsp}
    - Bluetooth: Fix race condition in hidp_session_thread
    - bluetooth: btbcm: Fix logic error in forming the board name.
    - Bluetooth: Free potentially unfreed SCO connection
    - Bluetooth: hci_conn: Fix possible UAF
    - btrfs: restore the thread_pool=...

[Christian Ehrhardt  (paelzer)]

Hi,
I was rechecking the numbers after a while as you called this fix released and confirmed for some more cleanup in mantic. AFAIC even Lunar didn't improve.

To better see it I'll attach an image of daily metrics, it didn't recover since my report on early April and Lunar as well as Mantic are still quite high compared to former releases.

I'll bump prio up to reflect that

[Dimitri John Ledkov (xnox)]

* Kernel 6.1 bumped the disk consumption on default images by 15%
    (LP: #2015867)
    - [Packaging] introduce a separate linux-lib-rust package

is fixed in mantic-proposed, this bug will close once this bug report migrates.

[Christian Ehrhardt  (paelzer)]

Seen the first lunar test with the fix.
Overall we now have:
 - 859 mb (before) -> 984 mb (bug) -> 899 mb (fixed)

So there would be more to get back to the old sizes, but I've understood that this seemed to be what can be done for now.

I'll update again to confirm once I see the change land in Mantic.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.3.0-7.7

---------------
linux (6.3.0-7.7) mantic; urgency=medium

  * mantic/linux: 6.3.0-7.7 -proposed tracker (LP: #2023297)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/master)

 -- Paolo Pisati <email address hidden> Thu, 08 Jun 2023 16:44:41 +0200

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-6.5/6.5.0-1007.7~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-done-jammy-linux-azure-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-failed-jammy-linux-azure-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-6.5/6.5.0-1008.8~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-done-jammy-linux-aws-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-failed-jammy-linux-aws-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!