ubuntu_lttng_smoke_test crash with B-5.0 AWS/Azure/GCP
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ubuntu-kernel-tests |
Fix Released
|
Undecided
|
Unassigned | ||
linux-aws-edge (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Bionic |
Invalid
|
Undecided
|
Unassigned | ||
lttng-modules (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Kleber Sacilotto de Souza |
Bug Description
[Impact]
Test will crash and hang, until timeout on jenkins.
This test passed with B-4.15 AWS, and D-5.0 AWS, just not B-5.0 AWS
There is no report and no obvious failure on the jenkins output (just marked as build aborted), this can only be found by tailing the syslog when you run the test.
Test output:
Running '/home/
[stdout] == lttng smoke test of session create/destroy ==
[stdout] Session test-kernel-session created.
[stdout] Traces will be written in /tmp/lttng-
[stdout] PASSED (lttng create)
[stdout] Session test-kernel-session destroyed
[stdout] PASSED (lttng destroy)
[stdout]
[stdout] == lttng smoke test trace context switches ==
[stdout] Session test-kernel-session created.
[stdout] Traces will be written in /tmp/lttng-
[stdout] PASSED (lttng create)
syslog output:
kernel: [ 253.282158] lttng_kretprobes: loading out-of-tree module taints kernel.
kernel: [ 253.282191] lttng_kretprobes: module verification failed: signature and/or required key missing - tainting kernel
kernel: [ 253.293676] BUG: unable to handle kernel paging request at 0000000000025534
kernel: [ 253.296613] #PF error: [normal kernel read fault]
kernel: [ 253.298623] PGD 0 P4D 0
kernel: [ 253.299783] Oops: 0000 [#1] SMP PTI
kernel: [ 253.301313] CPU: 3 PID: 955 Comm: lttng-sessiond Tainted: G OE 5.0.0-1014-aws #16~18.04.1-Ubuntu
kernel: [ 253.305391] Hardware name: Amazon EC2 m5.xlarge/, BIOS 1.0 10/16/2017
kernel: [ 253.308102] RIP: 0010:lttng_
kernel: [ 253.310886] Code: eb 1a 49 39 c6 0f 85 ab 00 00 00 49 8b 55 10 41 83 c4 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 4c 63 34 82 <49> 8b 1e 48 89 df e8 33 fb ff ff 48 85 c0 49 89 c7 74 52 49 8b 47
kernel: [ 253.318387] RSP: 0018:ffffb9c802
kernel: [ 253.320596] RAX: 0000000000000000 RBX: ffff9a27cbec9c00 RCX: 0000000000000041
kernel: [ 253.323556] RDX: ffffffffc0456a0c RSI: 0000000000000001 RDI: ffffffffc0898100
kernel: [ 253.326507] RBP: ffffb9c80232fc08 R08: ffff9a27d2ba70c0 R09: ffff9a27d2403680
kernel: [ 253.329460] R10: ffffb9c80232fb48 R11: 00000000ffffffff R12: 0000000000000000
kernel: [ 253.332420] R13: ffff9a27cbec9c00 R14: 0000000000025534 R15: ffffffffc0898740
kernel: [ 253.335375] FS: 00007f26e1d4970
kernel: [ 253.338700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 253.341118] CR2: 0000000000025534 CR3: 000000040c86c003 CR4: 00000000007606e0
kernel: [ 253.345464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [ 253.349798] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: [ 253.354126] PKRU: 55555554
kernel: [ 253.356719] Call Trace:
kernel: [ 253.359210] register_
kernel: [ 253.362856] ? 0xffffffffc08d7000
kernel: [ 253.365675] lttng_tracepoin
kernel: [ 253.369380] lttng_events_
kernel: [ 253.372952] ? 0xffffffffc08d7000
kernel: [ 253.375739] do_one_
kernel: [ 253.378733] ? _cond_resched+
kernel: [ 253.381693] ? kmem_cache_
kernel: [ 253.385037] do_init_
kernel: [ 253.387999] load_module+
kernel: [ 253.390956] __do_sys_
kernel: [ 253.394153] ? __do_sys_
kernel: [ 253.397416] __x64_sys_
kernel: [ 253.400618] do_syscall_
kernel: [ 253.403547] entry_SYSCALL_
kernel: [ 253.407035] RIP: 0033:0x7f26e738d839
kernel: [ 253.409949] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1f f6 2c 00 f7 d8 64 89 01 48
kernel: [ 253.421539] RSP: 002b:00007f26e1
kernel: [ 253.427373] RAX: ffffffffffffffda RBX: 00007f26cc004d70 RCX: 00007f26e738d839
kernel: [ 253.431692] RDX: 0000000000000000 RSI: 00007f26e8d0a145 RDI: 0000000000000030
kernel: [ 253.435978] RBP: 00007f26e8d0a145 R08: 0000000000000000 R09: 00007f26cc004b00
kernel: [ 253.440270] R10: 0000000000000030 R11: 0000000000000246 R12: 0000000000000000
kernel: [ 253.444581] R13: 00007f26cc004570 R14: 0000000000000000 R15: 00007f26cc004b00
kernel: [ 253.448853] Modules linked in: lttng_tracer(OE+) lttng_statedump(OE) lttng_kprobes(OE) lttng_clock(OE) lttng_lib_
kernel: [ 253.478768] CR2: 0000000000025534
kernel: [ 253.481575] ---[ end trace 9c47de0da4f80343 ]---
kernel: [ 253.484885] RIP: 0010:lttng_
kernel: [ 253.489142] Code: eb 1a 49 39 c6 0f 85 ab 00 00 00 49 8b 55 10 41 83 c4 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 4c 63 34 82 <49> 8b 1e 48 89 df e8 33 fb ff ff 48 85 c0 49 89 c7 74 52 49 8b 47
kernel: [ 253.500676] RSP: 0018:ffffb9c802
kernel: [ 253.504223] RAX: 0000000000000000 RBX: ffff9a27cbec9c00 RCX: 0000000000000041
kernel: [ 253.508498] RDX: ffffffffc0456a0c RSI: 0000000000000001 RDI: ffffffffc0898100
kernel: [ 253.512783] RBP: ffffb9c80232fc08 R08: ffff9a27d2ba70c0 R09: ffff9a27d2403680
kernel: [ 253.517125] R10: ffffb9c80232fb48 R11: 00000000ffffffff R12: 0000000000000000
kernel: [ 253.521421] R13: ffff9a27cbec9c00 R14: 0000000000025534 R15: ffffffffc0898740
kernel: [ 253.525702] FS: 00007f26e1d4970
kernel: [ 253.531708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 253.535444] CR2: 0000000000025534 CR3: 000000040c86c003 CR4: 00000000007606e0
kernel: [ 253.539722] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [ 253.544020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: [ 253.548300] PKRU: 55555554
I am proposing we backport the lttng-modules from Disco (2.10.8-1ubuntu1) to fix the issues with most recent kernel.
[Test Case]
- Install lttng-modules-dkms
- load lttng-tracer module
I have built the package on bionic and load-tested with linux 4.15 and some 5.0 backports.
[Regression Potential]
Backporting a newer version to Bionic has the potential to break compilation or users with not up-to-date 4.15 kernels.
description: | updated |
Changed in linux-aws-edge (Ubuntu): | |
status: | New → Invalid |
Changed in linux-aws-edge (Ubuntu Bionic): | |
status: | New → Invalid |
Changed in lttng-modules (Ubuntu): | |
status: | New → In Progress |
Changed in lttng-modules (Ubuntu Bionic): | |
status: | New → In Progress |
description: | updated |
On generic B-5.0 kernel, this test just failed with launching session daemon:
== lttng smoke test of session create/destroy == lttng-sessiond)
Error: Session daemon terminated with an error (exit status: 1)
Error: Problem occurred while launching session daemon (/usr/bin/
Error: Command error
Spawning a session daemon
FAILED (lttng create)
Error: No session daemon is available
Error: Command error
FAILED (lttng destroy)
== lttng smoke test trace context switches == lttng-sessiond)
Error: Session daemon terminated with an error (exit status: 1)
Error: Problem occurred while launching session daemon (/usr/bin/
Error: Command error
Spawning a session daemon
FAILED (lttng create)