| 2022-09-29 10:34:45 |
Po-Hsu Lin |
description |
While closing bug 1906389, I found this issue with Jammy 5.15.0-48-generic on P9 node baltar
We're not running this test on PowerPC, maybe we should consider to cover this.
The ubuntu_lttng_smoke_test will hang with:
Running '/home/ubuntu/autotest/client/tests/ubuntu_lttng_smoke_test/ubuntu_lttng_smoke_test.sh'
== lttng smoke test of session create/destroy ==
Session test-kernel-session created.
Traces will be output to /tmp/lttng-kernel-trace-12196-session
PASSED (lttng create)
Session test-kernel-session destroyed
PASSED (lttng destroy)
== lttng smoke test trace context switches ==
Session test-kernel-session created.
Traces will be output to /tmp/lttng-kernel-trace-12196-session
PASSED (lttng create)
(Test hang here)
In dmesg you will see:
[ 6804.283169] lttng_kretprobes: loading out-of-tree module taints kernel.
[ 6804.283265] lttng_kretprobes: module verification failed: signature and/or required key missing - tainting kernel
[ 6804.434672] BUG: Unable to handle kernel data access on read at 0xc0080000129f0000
[ 6804.434704] Faulting instruction address: 0xc0000000002a998c
[ 6804.434728] Oops: Kernel access of bad area, sig: 11 [#1]
[ 6804.434740] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA PowerNV
[ 6804.434774] Modules linked in: lttng_tracer(OE+) lttng_statedump(OE) lttng_wrapper(OE) lttng_uprobes(OE) lttng_clock(OE) lttng_kprobes(OE) lttng_lib_ring_buffer(OE) lttng_kretprobes(OE) tls joydev input_leds mac_hid ofpart cmdlinepart powernv_flash at24 mtd opal_prd uio_pdrv_genirq ibmpowernv uio ipmi_powernv vmx_crypto sch_fq_codel dm_multipath pstore_blk scsi_dh_rdac scsi_dh_emc ipmi_devintf pstore_zone ramoops ipmi_msghandler scsi_dh_alua reed_solomon ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ses enclosure scsi_transport_sas usbhid hid ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec rc_core crct10dif_vpmsum crc32c_vpmsum drm xhci_pci i40e aacraid xhci_pci_renesas drm_panel_orientation_quirks
[ 6804.435085] CPU: 74 PID: 4534 Comm: Client manageme Tainted: G OE 5.15.0-48-generic #54-Ubuntu
[ 6804.435124] NIP: c0000000002a998c LR: c0000000002a9fd0 CTR: c0000000002a9e2c
[ 6804.435158] REGS: c0000000093cb670 TRAP: 0300 Tainted: G OE (5.15.0-48-generic)
[ 6804.435193] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 84228284 XER: 20040151
[ 6804.435245] CFAR: c0000000002a9fcc DAR: c0080000129f0000 DSISR: 40000000 IRQMASK: 0
GPR00: c0000000002a9fd0 c0000000093cb910 c008000013f58100 00000000001062b8
GPR04: c0000000093cb938 c0080000128e9d48 0000000000000080 0000000000000000
GPR08: ffffffffffffff99 c0080000128e8100 00000000001062b9 c008000012a77ae8
GPR12: c008000012a77ea8 c000000ffff8f100 0000000020000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 000007f1831cbec8 000007f1831cbe48
GPR20: 000007f1831cbf28 00007e1e2c006d10 000007f1831c2478 00007e1e2c003190
GPR24: 0000000000000000 0000000000000000 000000000004ac15 00000000001062b8
GPR28: c008000013f40158 c008000013f40198 0000000000000000 000000000009e2b8
[ 6804.435518] NIP [c0000000002a998c] kallsyms_expand_symbol.constprop.0+0x1c/0xd0
[ 6804.435556] LR [c0000000002a9fd0] kallsyms_lookup_name+0x1b0/0x2b0
[ 6804.435590] Call Trace:
[ 6804.435607] [c0000000093cb910] [c0000000002f5394] unregister_kprobe+0x64/0xb0 (unreliable)
[ 6804.435645] [c0000000093cb9f0] [c008000013f302d4] wrapper_kallsyms_lookup_name+0x4c/0xf0 [lttng_wrapper]
[ 6804.435700] [c0000000093cba60] [c008000013f300b0] wrapper_get_pfnblock_flags_mask_init+0x28/0x60 [lttng_wrapper]
[ 6804.435749] [c0000000093cba80] [c008000014a5ead0] lttng_events_init+0x40/0x344 [lttng_tracer]
[ 6804.435884] [c0000000093cbb20] [c000000000012320] do_one_initcall+0x60/0x2d0
[ 6804.435920] [c0000000093cbc00] [c0000000002a5888] do_init_module+0x68/0x340
[ 6804.435955] [c0000000093cbc90] [c0000000002a84b4] __do_sys_finit_module+0xd4/0x160
[ 6804.435992] [c0000000093cbdb0] [c00000000002f764] system_call_exception+0x114/0x2c0
[ 6804.436028] [c0000000093cbe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
[ 6804.436069] --- interrupt: 3000 at 0x7e1e45445fc0
[ 6804.436091] NIP: 00007e1e45445fc0 LR: 0000000000000000 CTR: 0000000000000000
[ 6804.436125] REGS: c0000000093cbe80 TRAP: 3000 Tainted: G OE (5.15.0-48-generic)
[ 6804.436160] MSR: 900000000280f033 <SF,HV,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 44228248 XER: 00000000
[ 6804.436211] IRQMASK: 0
GPR00: 0000000000000161 00007e1e327b8c90 00007e1e45566e00 0000000000000048
GPR04: 00007e1e45a1b530 0000000000000000 0000000000000048 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007e1e327d5b00 0000000020000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 000007f1831cbec8 000007f1831cbe48
GPR20: 000007f1831cbf28 00007e1e2c006d10 000007f1831c2478 00007e1e2c003190
GPR24: 0000000000000000 00007e1e2c007410 00007e1e327b8fb0 00007e1e2c0073c0
GPR28: 00007e1e45a1b530 0000000000000000 00007e1e2c007410 00007e1e2c003420
[ 6804.436425] NIP [00007e1e45445fc0] 0x7e1e45445fc0
[ 6804.436459] LR [0000000000000000] 0x0
[ 6804.436477] --- interrupt: 3000
[ 6804.436496] Instruction dump:
[ 6804.436515] 48349e9d 60000000 4bffff84 60000000 60420000 3c4c0281 3842ac90 7c0802a6
[ 6804.436557] 60000000 3d22fe99 39430001 38a91c48 <7d2518ae> 7ca32a14 2c090000 7c695214
[ 6804.436601] ---[ end trace c14a6519d9a9d0c6 ]--- |
While closing bug 1906389, I found this issue with Jammy 5.15.0-48-generic on P9 node baltar
We're not running this test on PowerPC, maybe we should consider to cover this.
The ubuntu_lttng_smoke_test will hang with:
Running '/home/ubuntu/autotest/client/tests/ubuntu_lttng_smoke_test/ubuntu_lttng_smoke_test.sh'
== lttng smoke test of session create/destroy ==
Session test-kernel-session created.
Traces will be output to /tmp/lttng-kernel-trace-12196-session
PASSED (lttng create)
Session test-kernel-session destroyed
PASSED (lttng destroy)
== lttng smoke test trace context switches ==
Session test-kernel-session created.
Traces will be output to /tmp/lttng-kernel-trace-12196-session
PASSED (lttng create)
(Test hang here)
When trying to load the lttng_tracer module, you will see the following error in dmesg:[ 310.673923] lttng_kretprobes: loading out-of-tree module taints kernel.
[ 310.674033] lttng_kretprobes: module verification failed: signature and/or required key missing - tainting kernel
[ 310.826562] BUG: Unable to handle kernel data access at 0xfffffffffffffff8
[ 310.826594] Faulting instruction address: 0xc0000000002a8ca0
[ 310.826618] Oops: Kernel access of bad area, sig: 11 [#1]
[ 310.826631] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA PowerNV
[ 310.826647] Modules linked in: lttng_tracer(OE+) lttng_statedump(OE) lttng_wrapper(OE) lttng_uprobes(OE) lttng_clock(OE) lttng_kprobes(OE) lttng_lib_ring_buffer(OE) lttng_kretprobes(OE) ofpart cmdlinepart powernv_flash mtd at24 uio_pdrv_genirq opal_prd ibmpowernv uio joydev input_leds mac_hid ipmi_powernv vmx_crypto sch_fq_codel ipmi_devintf ramoops pstore_blk dm_multipath ipmi_msghandler scsi_dh_rdac scsi_dh_emc pstore_zone scsi_dh_alua reed_solomon ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic ses enclosure scsi_transport_sas usbhid hid ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec rc_core drm crct10dif_vpmsum crc32c_vpmsum xhci_pci i40e aacraid xhci_pci_renesas drm_panel_orientation_quirks
[ 310.826957] CPU: 78 PID: 9462 Comm: modprobe Tainted: G OE 5.15.0-48-generic #54-Ubuntu
[ 310.826995] NIP: c0000000002a8ca0 LR: c0000000002a8c34 CTR: 0000000000000011
[ 310.827030] REGS: c000000006c87630 TRAP: 0380 Tainted: G OE (5.15.0-48-generic)
[ 310.827066] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44222282 XER: 20040151
[ 310.827109] CFAR: c0000000002a8c3c IRQMASK: 0
GPR00: c0000000002aa078 c000000006c878d0 c008000013ad8100 0000000000000000
GPR04: 000000000000001c 000000000000003a 0000000024222282 c0000000530df400
GPR08: c000000006c87978 c0080000138c8100 0000000000000000 0000000000002000
GPR12: c0000000002a9e2c c000000ffff8ab00 0000064758594540 0000064758590f18
GPR16: 0000000000000000 0000000000000000 0000000000000548 0000064758594548
GPR20: 0000064758591970 000006475368dea8 0000064758595940 0000064758590eb0
GPR24: 0000000000000000 0000000000000000 000000000000000a 0000000000000000
GPR28: c008000013ac0158 c0080000138c7268 c008000013ac0158 fffffffffffffff8
[ 310.827386] NIP [c0000000002a8ca0] module_kallsyms_lookup_name+0xa0/0x110
[ 310.827423] LR [c0000000002a8c34] module_kallsyms_lookup_name+0x34/0x110
[ 310.827457] Call Trace:
[ 310.827475] [c000000006c87910] [c0000000002aa078] kallsyms_lookup_name+0x258/0x2b0
[ 310.827512] [c000000006c879f0] [c008000013ab02d4] wrapper_kallsyms_lookup_name+0x4c/0xf0 [lttng_wrapper]
[ 310.827553] [c000000006c87a60] [c008000013ab00b0] wrapper_get_pfnblock_flags_mask_init+0x28/0x60 [lttng_wrapper]
[ 310.827609] [c000000006c87a80] [c008000014fcead0] lttng_events_init+0x40/0x344 [lttng_tracer]
[ 310.827781] [c000000006c87b20] [c000000000012320] do_one_initcall+0x60/0x2d0
[ 310.827819] [c000000006c87c00] [c0000000002a5888] do_init_module+0x68/0x340
[ 310.827859] [c000000006c87c90] [c0000000002a84b4] __do_sys_finit_module+0xd4/0x160
[ 310.827897] [c000000006c87db0] [c00000000002f764] system_call_exception+0x114/0x2c0
[ 310.827938] [c000000006c87e10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
[ 310.827981] --- interrupt: 3000 at 0x794375935fc0
[ 310.828017] NIP: 0000794375935fc0 LR: 0000000000000000 CTR: 0000000000000000
[ 310.828051] REGS: c000000006c87e80 TRAP: 3000 Tainted: G OE (5.15.0-48-generic)
[ 310.828102] MSR: 900000000280f033 <SF,HV,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 44222248 XER: 00000000
[ 310.828158] IRQMASK: 0
GPR00: 0000000000000161 00007fffefcf5db0 0000794375a56e00 000000000000000a
GPR04: 0000064753690b40 0000000000000000 000000000000000a 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007943761eb9c0 0000064758594540 0000064758590f18
GPR16: 0000000000000000 0000000000000000 0000000000000548 0000064758594548
GPR20: 0000064758591970 000006475368dea8 0000064758595940 0000064758590eb0
GPR24: 0000000000000000 00000647536903f0 0000064753690b40 00000647585905f0
GPR28: 0000064758590f30 0000000000040000 0000064758590eb0 0000064758595b20
[ 310.828394] NIP [0000794375935fc0] 0x794375935fc0
[ 310.828406] LR [0000000000000000] 0x0
[ 310.828417] --- interrupt: 3000
[ 310.828426] Instruction dump:
[ 310.828436] 7c0803a6 4e800020 60000000 60420000 3d22ffdf fba10028 3fa2ffdf ebe9f168
[ 310.828480] 3bbdf168 7c3fe800 3bfffff8 41820034 <813f0000> 7fe3fb78 7fc4f378 2c090003
[ 310.828536] ---[ end trace ae576cf495bc78dd ]--- |
|
