ubuntu_lttng_smoke_test failed on Azure 4.15

Bug #1778642 reported by Po-Hsu Lin on 2018-06-26
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Undecided
Unassigned
linux-azure (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
lttng-modules (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

SRU Request:

[Impact]

After bionic/linux-azure commit 4fbb658dd8ce96384852ceae18b0b09034de7b36 (UBUNTU: [Config] azure: CONFIG_HOTPLUG_CPU=n) the lttng_tracer module cannot find some symbols and cannot be loaded.

 kernel: [ 639.538251] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.538280] lttng_tracer: Unknown symbol lttng_hp_online (err 0)

This is caused by lttng-modules incorrectly handling the case when CONFIG_HOTPLUG_CPU is not enabled.

[Test Case]

Install linux-azure >= 4.15.0-1014, install lttng-modules-dkms and run the autotest ubuntu_lttng_smoke_test testcase (or just 'sudo lttng list --kernel').

[Fix]

commit 894ce5dc825733d1ccab25b1aede9ce0be2a7c4e (Fix: do not use CONFIG_HOTPLUG_CPU for the new hotplug API)

commit 1ee63d8d1b0aaf30564d3879d8e9282d0ddfef1b (Cleanup: comment about CONFIG_HOTPLUG_CPU ifdef)

[Regression Potential]

Very low. The two patches can be cleanly cherry-picked from the upstream commits. The first one that really fixed the issue just changes a single #ifdef to compile a piece of code when CONFIG_HOTPLUG_CPU is not defined. The second one just updates some comments on the code to make them consistent with the changes.

[Original Description]

This test will pass with the older kernel, 4.15.0-1013-azure, but not the 4.15.0-1014-azure in -proposed.

It looks like the kernel module can be installed, but four tests have failed:

== lttng smoke test of session create/destroy ==
Session test-kernel-session created.
Traces will be written in /tmp/lttng-kernel-trace-14716-session
PASSED (lttng create)
Session test-kernel-session destroyed
PASSED (lttng destroy)

== lttng smoke test list kernel events ==
FAILED (lttng list --kernel)
FAILED (lttng list --kernel more output expected)

== lttng smoke test trace open/close system calls ==
Session test-kernel-session created.
Traces will be written in /tmp/lttng-kernel-trace-14716-session
PASSED (lttng create)
FAILED (lttng enable-event)
Session test-kernel-session destroyed

== lttng smoke test trace context switches ==
Session test-kernel-session created.
Traces will be written in /tmp/lttng-kernel-trace-14716-session
PASSED (lttng create)
FAILED (lttng enable-event)
Session test-kernel-session destroyed

Summary: 4 passed, 4 failed

http://10.246.72.46/4.15.0-1014.14-azure/bionic-linux-azure-azure-4.15.0-Standard_D12-ubuntu_lttng_smoke_test/ubuntu_lttng_smoke_test/results/ubuntu_lttng_smoke_test.lttng-smoke-test/debug/ubuntu_lttng_smoke_test.lttng-smoke-test.DEBUG.html

Error message from /var/log/syslog:

 kernel: [ 639.531889] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.538251] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.538280] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.606845] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.613331] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.613360] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.677124] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.680857] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.680886] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.737914] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.753366] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.753394] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.829710] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.843736] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.843765] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available

Po-Hsu Lin (cypressyew) on 2018-06-27
description: updated
Po-Hsu Lin (cypressyew) wrote :

Comment #1 is invalid, this test does not work on xenial/Azure because of the module build issue (bug 1765673)

This is probably being caused by the following commit:

commit 4fbb658dd8ce96384852ceae18b0b09034de7b36
Author: Marcelo Henrique Cerri <email address hidden>
Date: Mon Jun 11 18:22:54 2018 -0300

    UBUNTU: [Config] azure: CONFIG_HOTPLUG_CPU=n

The lttng_hp_prepare and lttng_hp_online variables are declared inside a #ifdef block and are probably used somewhere else without the proper checking.

So this is a regression that needs to be fixed on the lttng dkms code.

This bug seems to be resolved by the following commits from lttng-modules upstream:

===============================================
commit 1ee63d8d1b0aaf30564d3879d8e9282d0ddfef1b
Author: Michael Jeanson <email address hidden>
Date: Tue Mar 13 12:14:43 2018 -0400

    Cleanup: comment about CONFIG_HOTPLUG_CPU ifdef

    Signed-off-by: Michael Jeanson <email address hidden>
    Signed-off-by: Mathieu Desnoyers <email address hidden>

commit 894ce5dc825733d1ccab25b1aede9ce0be2a7c4e
Author: Lars Persson <email address hidden>
Date: Sun Mar 11 15:02:43 2018 +0100

    Fix: do not use CONFIG_HOTPLUG_CPU for the new hotplug API

    Kernel configurations without CONFIG_HOTPLUG_CPU throw an unknown
    symbol error when attempting to insert the lttng-trace module:
     lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
     lttng_tracer: Unknown symbol lttng_hp_online (err 0)

    This was caused by lttng-events and lttng-context-perf-counter not
    agreeing on which preprocessor condition that should guard the use of
    the hotplug API. In fact the API is available also on kernels built
    without CONFIG_HOTPLUG_CPU.

    Signed-off-by: Lars Persson <email address hidden>
    Signed-off-by: Mathieu Desnoyers <email address hidden>
===============================================

I will build a test package and attach here a debdiff.

Changed in linux-azure (Ubuntu):
status: New → Invalid
Changed in ubuntu-kernel-tests:
status: New → Invalid
Changed in linux-azure (Ubuntu Bionic):
status: New → Invalid
Changed in lttng-modules (Ubuntu Bionic):
status: New → In Progress
Changed in lttng-modules (Ubuntu):
status: New → Fix Released
Changed in lttng-modules (Ubuntu Cosmic):
status: Fix Released → In Progress
Changed in linux-azure (Ubuntu Xenial):
status: New → Invalid
Changed in lttng-modules (Ubuntu Xenial):
status: New → In Progress
tags: added: patch
description: updated
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lttng-modules - 2.10.5-1ubuntu3

---------------
lttng-modules (2.10.5-1ubuntu3) cosmic; urgency=medium

  * ubuntu_lttng_smoke_test failed on Azure 4.15 (LP: #1778642)
    - debian/patches/0009-Fix-do-not-use-CONFIG_HOTPLUG_CPU-for-the-new-hotplu.patch
    - debian/patches/0010-Cleanup-comment-about-CONFIG_HOTPLUG_CPU-ifdef.patch

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 19 Jul 2018 14:34:36 +0200

Changed in lttng-modules (Ubuntu Cosmic):
status: In Progress → Fix Released

Hello Po-Hsu, or anyone else affected,

Accepted lttng-modules into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.8.0-1ubuntu1~16.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lttng-modules (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Po-Hsu Lin (cypressyew) wrote :

Verified with Xenial Azure cloud, it works as expected.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
no longer affects: lttng-modules

The verification of the Stable Release Update for lttng-modules has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lttng-modules - 2.8.0-1ubuntu1~16.04.7

---------------
lttng-modules (2.8.0-1ubuntu1~16.04.7) xenial; urgency=medium

  * ubuntu_lttng_smoke_test failed on Azure 4.15 (LP: #1778642)
    - debian/patches/0041-Fix-do-not-use-CONFIG_HOTPLUG_CPU-for-the-new-hotplu.patch
    - debian/patches/0042-Cleanup-comment-about-CONFIG_HOTPLUG_CPU-ifdef.patch

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 19 Jul 2018 17:47:16 +0200

Changed in lttng-modules (Ubuntu Xenial):
status: Fix Committed → Fix Released

Hello Po-Hsu, or anyone else affected,

Accepted lttng-modules into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.10.5-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lttng-modules (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done

Verified the fix on Bionic with linux-image-4.15.0-1018-azure=4.15.0-1018.18 and lttng-modules-dkms=2.10.5-1ubuntu1.1. The fix works as expected and ubuntu_lttng_smoke_test testcase passes.

tags: added: verification-done-bionic
removed: verification-needed-bionic
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lttng-modules - 2.10.5-1ubuntu1.1

---------------
lttng-modules (2.10.5-1ubuntu1.1) bionic; urgency=medium

  * ubuntu_lttng_smoke_test failed on Azure 4.15 (LP: #1778642)
    - debian/patches/0003-Fix-do-not-use-CONFIG_HOTPLUG_CPU-for-the-new-hotplu.patch
    - debian/patches/0004-Cleanup-comment-about-CONFIG_HOTPLUG_CPU-ifdef.patch

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 19 Jul 2018 12:49:06 +0200

Changed in lttng-modules (Ubuntu Bionic):
status: Fix Committed → Fix Released
Po-Hsu Lin (cypressyew) on 2018-12-05
Changed in ubuntu-kernel-tests:
status: Invalid → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers