ubuntu_lttng_smoke_test failed with module build on J-5.15.0-49.55

[Po-Hsu Lin]

[SRU Justification]

Impact: Upstream stable backported a patch from 5.17 into 5.15.58 which allows to pass a reason message when dropping network SKBs. This however requires some adaptions to lttng-modules (again) to handle the modified interface.

Fix: Picking 3 patches from upstream lttng-modules which handle the changed interface as well as add the required code to enable that change for 5.15 stable kernels starting with subversion 58.

Testcase: Compiling for the latest 5.15 Ubuntu kernels fails with below messages. Once the 3 fixes are applied this succeeds (and the previous kernel also still works).

Regression potential: Probes related to dropping SKBs might show issues. But also tooling which uses these might get surprised by suddenly seeing reason messages.

---

Issue found on Jammy 5.15.0-49.55

Module build log:
 Running 'cat /var/lib/dkms/lttng-modules/*/build/make.log'
 DKMS make.log for lttng-modules-2.13.1 for kernel 5.15.0-49-generic (x86_64)
 Tue Sep 20 04:39:58 UTC 2022
 make: Entering directory '/usr/src/linux-headers-5.15.0-49-generic'
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-discard.o
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:62: File ./arch/x86/kvm/lapic.h not found. Probe "kvm" x86-specific is disabled. Use full kernel source tree to enable it.
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/probes/lttng-test.o
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:166: Files ./fs/btrfs/*.h not found. Probe "btrfs" is disabled. Use full kernel source tree to enable it.
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:177: Files ./fs/ext4/*.h not found. Probe "ext4" is disabled. Use full kernel source tree to enable it.
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:206: File ./drivers/base/regmap/trace.h not found. Probe "regmap" is disabled. Need Linux 4.1+ kernel source tree to enable it.
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_backend.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-sched.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/clock-plugin/lttng-clock-plugin-test.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-overwrite.o
   LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/lttng-test.o
   LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/lttng-clock-plugin-test.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-metadata-client.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_frontend.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-irq.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-mmap-discard.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-mmap-overwrite.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-timer.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_iterator.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_vfs.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_splice.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-kmem.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_mmap.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/prio_heap/lttng_prio_heap.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/../wrapper/splice.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/counter/counter.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-module.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-power.o
   LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/lttng-lib-ring-buffer.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-statedump.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-9p.o
   LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/lttng-counter.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-metadata-mmap-client.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-i2c.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-kvm.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-x86-irq-vectors.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-x86-exceptions.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-signal.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-block.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-napi.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-skb.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-net.o
 In file included from /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/../../include/lttng/define_trace.h:87,
                  from /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/../../include/instrumentation/events/skb.h:59,
                  from /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-skb.c:29:
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/../../include/lttng/tracepoint-event-impl.h:133:6: error: conflicting types for ‘trace_kfree_skb’; have ‘void(struct sk_buff *, void *)’
   133 | void trace_##_name(_proto);
       | ^~~~~~
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/../../include/lttng/tracepoint-event-impl.h:45:9: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP’
    45 | LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP(map, name, map, PARAMS(proto), PARAMS(args))
       | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/../../include/instrumentation/events/skb.h:16:1: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_MAP’
    16 | LTTNG_TRACEPOINT_EVENT_MAP(kfree_skb,
       | ^~~~~~~~~~~~~~~~~~~~~~~~~~
 In file included from ./include/trace/events/skb.h:10,
                  from /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-skb.c:18:
 ./include/linux/tracepoint.h:245:28: note: previous definition of ‘trace_kfree_skb’ with type ‘void(struct sk_buff *, void *, enum skb_drop_reason)’
   245 | static inline void trace_##name(proto) \
       | ^~~~~~
 ./include/linux/tracepoint.h:419:9: note: in expansion of macro ‘__DECLARE_TRACE’
   419 | __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
       | ^~~~~~~~~~~~~~~
 ./include/linux/tracepoint.h:553:9: note: in expansion of macro ‘DECLARE_TRACE’
   553 | DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
       | ^~~~~~~~~~~~~
 ./include/trace/events/skb.h:44:1: note: in expansion of macro ‘TRACE_EVENT’
    44 | TRACE_EVENT(kfree_skb,
       | ^~~~~~~~~~~
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-sock.o
 make[3]: *** [scripts/Makefile.build:297: /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-skb.o] Error 1
 make[3]: *** Waiting for unfinished jobs....
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-event-notifier-client.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-counter-client-percpu-32-modular.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-counter-client-percpu-64-modular.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-clock.o
   CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/msgpack/msgpack.o
 make[2]: *** [scripts/Makefile.build:560: /var/lib/dkms/lttng-modules/2.13.1/build/src/probes] Error 2
 make[2]: *** Waiting for unfinished jobs....
 make[1]: *** [scripts/Makefile.build:560: /var/lib/dkms/lttng-modules/2.13.1/build/src] Error 2
 make: *** [Makefile:1900: /var/lib/dkms/lttng-modules/2.13.1/build] Error 2
 make: Leaving directory '/usr/src/linux-headers-5.15.0-49-generic'

[Stefan Bader (smb)]

Will have to be done at least in focal as well to handle 5.15 backport kernels. Need to figure out the change which introduces this in case focal 5.4 kernels do the same. Then bionic needs a new lttng-modules as well.

[Stefan Bader (smb)]

Upstream stable included a backport which introduced kfree_skb_reason() to drop SKBs with a reason message. This requires changes to lttng-modules to handle v5.15.58+ kernels. The same changes will be needed in Focal for the 5.15 based backport kernels.

[Stefan Bader (smb)]

Test compile results in a VM with proposed lttng-modules against updates/proposed kernels in Jammy:

root@test-j1:/usr/src/lttng-modules-2.13.1# dpkg -l|grep lttng-modules
ii lttng-modules-dkms 2.13.1-1ubuntu0.22.04.2 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)
root@test-j1:/usr/src/lttng-modules-2.13.1# dkms status
lttng-modules/2.13.1, 5.15.0-48-generic, x86_64: installed
lttng-modules/2.13.1, 5.15.0-50-generic, x86_64: installed

[Stefan Bader (smb)]

This is the prepared change for Focal, but for testing we need a 5.15 kernel in proposed there (preferably hwe-5.15).

Additional note: the content of the patches is identical but the patched file is in a different location.

[Timo Aaltonen (tjaalton)]

Hello Po-Hsu, or anyone else affected,

Accepted lttng-modules into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.13.1-1ubuntu0.22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Po-Hsu, or anyone else affected,

Accepted lttng-modules into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.12.5-1ubuntu2~20.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Po-Hsu Lin (cypressyew)]

Verified on Jammy
ubuntu@glameow:~$ uname -a
Linux glameow 5.15.0-49-generic #55-Ubuntu SMP Fri Sep 16 16:29:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@glameow:~$ dpkg -l | grep lttng-module
ii lttng-modules-dkms 2.13.1-1ubuntu0.22.04.2 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)

[Po-Hsu Lin (cypressyew)]

Verified on Focal
ubuntu@curly:~$ uname -a
Linux curly 5.4.0-128-generic #144-Ubuntu SMP Tue Sep 20 11:00:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@curly:~$ dpkg -l | grep lttng-module
ii lttng-modules-dkms 2.12.5-1ubuntu2~20.04.4 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)

[Po-Hsu Lin (cypressyew)]

As smb pointed out, this module on Focal should be tested with Focal-5.15 instead.

I will revert the tags for Focal.

[Stefan Bader (smb)]

Both has value. Testing with 5.4 ensures it still works there and with latest focal -5.15 tells us it still works (again) with the changes made in 5.15 kernel side.

[Po-Hsu Lin (cypressyew)]

Verified with F-aws-5.15
$ uname -a
Linux ip-172-31-31-188 5.15.0-1021-aws #25~20.04.1-Ubuntu SMP Thu Sep 22 13:59:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ dpkg -l | grep lttng-module
ii lttng-modules-dkms 2.12.5-1ubuntu2~20.04.4 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)

F-hwe-5.15 is not ready yet, but I think maybe this is good enough?

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for lttng-modules has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lttng-modules - 2.12.5-1ubuntu2~20.04.4

---------------
lttng-modules (2.12.5-1ubuntu2~20.04.4) focal; urgency=medium

  * Backport fixes to handle kfree_skb_reason() in 5.15.y (LP: #1990232)
    - d/p/0010-compat-v5.15.58-add-kfree-skb-reason.patch
    - d/p/0011-compat-v5.15.58-rename-SKB-DROP-REASON-SOCKET-FILTER.patch
    - d/p/0012-compat-v5.15.58-introduce-kfree-skb-reason.patch

 -- Stefan Bader <email address hidden> Wed, 21 Sep 2022 16:32:14 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package lttng-modules - 2.13.1-1ubuntu0.22.04.2

---------------
lttng-modules (2.13.1-1ubuntu0.22.04.2) jammy; urgency=medium

  * Backport fixes to handle kfree_skb_reason() in 5.15.y (LP: #1990232)
    - d/p/fix-v5.15.58-add-kfree-skb-reason.patch
    - d/p/fix-v5.15.58-rename-SKB-DROP-REASON-SOCKET-FILTER.patch
    - d/p/fix-v5.15.58-introduce-kfree-skb-reason.patch

 -- Stefan Bader <email address hidden> Tue, 20 Sep 2022 16:26:36 +0200

[Po-Hsu Lin (cypressyew)]

Hints removed, closing this bug.