2019-07-23 08:08:27 |
Frank Heimes |
description |
== Comment: #0 - Robert J. Brenneman <rjbrenn@us.ibm.com> - 2019-05-30 11:16:45 ==
---Problem Description---
Kernel Oops 3b in libc-2.23 unable to handle pointer dereference in virtual kernel address space
Contact Information = rjbrenn@us.ibm.com
---uname output---
Linux ECOS0018 4.15.0-50-generic #54-Ubuntu SMP Tue May 7 05:57:08 UTC 2019 s390x s390x s390x GNU/Linux
Machine Type = z13 2964 NE1
---System Hang---
z/VM took a VMDUMP and reIPLed
---Debugger---
A debugger is not configured
---Steps to Reproduce---
boot system, start jenkins, let it run a couple days
Stack trace output:
05/29/19 13:24:06 Call Trace:
05/29/19 13:24:06 (?<000000000012b97a>? __tlb_remove_table+0x6a/0xd0)
05/29/19 13:24:06 ?<000000000012ba34>? tlb_remove_table_rcu+0x54/0x70
05/29/19 13:24:06 ?<00000000001f43b4>? rcu_process_callbacks+0x1d4/0x570
05/29/19 13:24:06 ?<00000000008e92d4>? __do_softirq+0x124/0x358
05/29/19 13:24:06 ?<0000000000179d52>? irq_exit+0xba/0xd0
05/29/19 13:24:06 ?<000000000010c412>? do_IRQ+0x8a/0xb8
05/29/19 13:24:06 ?<00000000008e87f0>? ext_int_handler+0x134/0x138
05/29/19 13:24:06 ?<0000000000102cee>? enabled_wait+0x4e/0xe0
05/29/19 13:24:06 (?<0000000000001201>? 0x1201)
05/29/19 13:24:06 ?<000000000010303a>? arch_cpu_idle+0x32/0x48
05/29/19 13:24:06 ?<00000000001c5ae8>? do_idle+0xe8/0x1a8
Oops output:
05/29/19 13:24:06 User process fault: interruption code 003b ilc:3 in libc-2.23.so?3ffaca00000+185000?
05/29/19 13:24:06 Failing address: 0000000000000000 TEID: 0000000000000800
05/29/19 13:24:06 Fault in primary space mode while using user ASCE.
05/29/19 13:24:06 AS:0000000710b241c7 R3:0000000000000024
05/29/19 13:24:06 Unable to handle kernel pointer dereference in virtual kernel address space
05/29/19 13:24:06 Failing address: 000003dbe0000000 TEID: 000003dbe0000403
05/29/19 13:24:06 Fault in home space mode while using kernel ASCE.
05/29/19 13:24:06 AS:0000000000ea8007 R3:0000000000000024
05/29/19 13:24:06 Oops: 003b ilc:3 ?#1? SMP
05/29/19 13:24:06 Modules linked in: veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_con
05/29/19 13:24:06 ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 dasd_fba_mod dasd_eckd_mod sha1_s390 sha_common dasd_mod
05/29/19 13:24:06 CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-50-generic #54-Ubuntu
05/29/19 13:24:06 Hardware name: IBM 2964 NE1 798 (z/VM 6.4.0)
05/29/19 13:24:06 Krnl PSW : 00000000dcb002be 0000000072762961 (__tlb_remove_table+0x56/0xd0)
05/29/19 13:24:06 R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
05/29/19 13:24:06 Krnl GPRS: ffffffffffffffba 000002b800000000 000002b800000003 0000000000eacac8
05/29/19 13:24:06 ffffffffffffffba 00000000000000b9 0700000000000000 000000000000000a
05/29/19 13:24:06 0404c00100000000 00000007d2fb5c38 00000007cf71fdf0 000003dbe0000000
05/29/19 13:24:06 000003dbe0000018 00000000008fe740 00000007cf71fd08 00000007cf71fcd8
05/29/19 13:24:06 Krnl Code: 000000000012b956: ec2c002a027f clij %r2,2,12,12b9aa
05/29/19 13:24:06 000000000012b95c: ec26001d037e cij %r2,3,6,12b996
05/29/19 13:24:06 #000000000012b962: 41c0b018 la %r12,24(%r11)
05/29/19 13:24:06 >000000000012b966: e548b0080000 mvghi 8(%r11),0
05/29/19 13:24:06 000000000012b96c: a7390008 lghi %r3,8
05/29/19 13:24:06 000000000012b970: b904002b lgr %r2,%r11
05/29/19 13:24:06 000000000012b974: c0e5000e8f8a brasl %r14,2fd888
05/29/19 13:24:06 000000000012b97a: a718ffff lhi %r1,-1
05/29/19 13:24:06 Call Trace:
05/29/19 13:24:06 (?<000000000012b97a>? __tlb_remove_table+0x6a/0xd0)
05/29/19 13:24:06 ?<000000000012ba34>? tlb_remove_table_rcu+0x54/0x70
05/29/19 13:24:06 ?<00000000001f43b4>? rcu_process_callbacks+0x1d4/0x570
05/29/19 13:24:06 ?<00000000008e92d4>? __do_softirq+0x124/0x358
05/29/19 13:24:06 ?<0000000000179d52>? irq_exit+0xba/0xd0
05/29/19 13:24:06 ?<000000000010c412>? do_IRQ+0x8a/0xb8
05/29/19 13:24:06 ?<00000000008e87f0>? ext_int_handler+0x134/0x138
05/29/19 13:24:06 ?<0000000000102cee>? enabled_wait+0x4e/0xe0
05/29/19 13:24:06 (?<0000000000001201>? 0x1201)
05/29/19 13:24:06 ?<000000000010303a>? arch_cpu_idle+0x32/0x48
05/29/19 13:24:06 ?<00000000001c5ae8>? do_idle+0xe8/0x1a8
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 01.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 04.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 05.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 00.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 06.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 02.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 03.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 ?<00000000001c5d86>? cpu_startup_entry+0x3e/0x48
05/29/19 13:24:06 ?<0000000000117240>? smp_start_secondary+0x120/0x140
05/29/19 13:24:06 ?<00000000008e8c46>? restart_int_handler+0x62/0x78
05/29/19 13:24:06 ?<0000000000000000>? (null)
05/29/19 13:24:06 Last Breaking-Event-Address:
05/29/19 13:24:06 ?<000000000012ba2e>? tlb_remove_table_rcu+0x4e/0x70
05/29/19 13:24:06
05/29/19 13:24:06 Kernel panic - not syncing: Fatal exception in interrupt
System Dump Location:
I will attach dumps here
*Additional Instructions for rjbrenn@us.ibm.com:
-Attach sysctl -a output output to the bug. |
== Comment: #0 - Robert J. Brenneman <rjbrenn@us.ibm.com> - 2019-05-30 11:16:45 ==
---Problem Description---
Kernel Oops 3b in libc-2.23 unable to handle pointer dereference in virtual kernel address space
Contact Information = rjbrenn@us.ibm.com
---uname output---
Linux ECOS0018 4.15.0-50-generic #54-Ubuntu SMP Tue May 7 05:57:08 UTC 2019 s390x s390x s390x GNU/Linux
Machine Type = z13 2964 NE1
---System Hang---
z/VM took a VMDUMP and reIPLed
(the attached and available dumps are Linux dumps)
---Debugger---
A debugger is not configured
---Steps to Reproduce---
boot system, start jenkins, let it run a couple days
Stack trace output:
05/29/19 13:24:06 Call Trace:
05/29/19 13:24:06 (?<000000000012b97a>? __tlb_remove_table+0x6a/0xd0)
05/29/19 13:24:06 ?<000000000012ba34>? tlb_remove_table_rcu+0x54/0x70
05/29/19 13:24:06 ?<00000000001f43b4>? rcu_process_callbacks+0x1d4/0x570
05/29/19 13:24:06 ?<00000000008e92d4>? __do_softirq+0x124/0x358
05/29/19 13:24:06 ?<0000000000179d52>? irq_exit+0xba/0xd0
05/29/19 13:24:06 ?<000000000010c412>? do_IRQ+0x8a/0xb8
05/29/19 13:24:06 ?<00000000008e87f0>? ext_int_handler+0x134/0x138
05/29/19 13:24:06 ?<0000000000102cee>? enabled_wait+0x4e/0xe0
05/29/19 13:24:06 (?<0000000000001201>? 0x1201)
05/29/19 13:24:06 ?<000000000010303a>? arch_cpu_idle+0x32/0x48
05/29/19 13:24:06 ?<00000000001c5ae8>? do_idle+0xe8/0x1a8
Oops output:
05/29/19 13:24:06 User process fault: interruption code 003b ilc:3 in libc-2.23.so?3ffaca00000+185000?
05/29/19 13:24:06 Failing address: 0000000000000000 TEID: 0000000000000800
05/29/19 13:24:06 Fault in primary space mode while using user ASCE.
05/29/19 13:24:06 AS:0000000710b241c7 R3:0000000000000024
05/29/19 13:24:06 Unable to handle kernel pointer dereference in virtual kernel address space
05/29/19 13:24:06 Failing address: 000003dbe0000000 TEID: 000003dbe0000403
05/29/19 13:24:06 Fault in home space mode while using kernel ASCE.
05/29/19 13:24:06 AS:0000000000ea8007 R3:0000000000000024
05/29/19 13:24:06 Oops: 003b ilc:3 ?#1? SMP
05/29/19 13:24:06 Modules linked in: veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_con
05/29/19 13:24:06 ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 dasd_fba_mod dasd_eckd_mod sha1_s390 sha_common dasd_mod
05/29/19 13:24:06 CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-50-generic #54-Ubuntu
05/29/19 13:24:06 Hardware name: IBM 2964 NE1 798 (z/VM 6.4.0)
05/29/19 13:24:06 Krnl PSW : 00000000dcb002be 0000000072762961 (__tlb_remove_table+0x56/0xd0)
05/29/19 13:24:06 R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
05/29/19 13:24:06 Krnl GPRS: ffffffffffffffba 000002b800000000 000002b800000003 0000000000eacac8
05/29/19 13:24:06 ffffffffffffffba 00000000000000b9 0700000000000000 000000000000000a
05/29/19 13:24:06 0404c00100000000 00000007d2fb5c38 00000007cf71fdf0 000003dbe0000000
05/29/19 13:24:06 000003dbe0000018 00000000008fe740 00000007cf71fd08 00000007cf71fcd8
05/29/19 13:24:06 Krnl Code: 000000000012b956: ec2c002a027f clij %r2,2,12,12b9aa
05/29/19 13:24:06 000000000012b95c: ec26001d037e cij %r2,3,6,12b996
05/29/19 13:24:06 #000000000012b962: 41c0b018 la %r12,24(%r11)
05/29/19 13:24:06 >000000000012b966: e548b0080000 mvghi 8(%r11),0
05/29/19 13:24:06 000000000012b96c: a7390008 lghi %r3,8
05/29/19 13:24:06 000000000012b970: b904002b lgr %r2,%r11
05/29/19 13:24:06 000000000012b974: c0e5000e8f8a brasl %r14,2fd888
05/29/19 13:24:06 000000000012b97a: a718ffff lhi %r1,-1
05/29/19 13:24:06 Call Trace:
05/29/19 13:24:06 (?<000000000012b97a>? __tlb_remove_table+0x6a/0xd0)
05/29/19 13:24:06 ?<000000000012ba34>? tlb_remove_table_rcu+0x54/0x70
05/29/19 13:24:06 ?<00000000001f43b4>? rcu_process_callbacks+0x1d4/0x570
05/29/19 13:24:06 ?<00000000008e92d4>? __do_softirq+0x124/0x358
05/29/19 13:24:06 ?<0000000000179d52>? irq_exit+0xba/0xd0
05/29/19 13:24:06 ?<000000000010c412>? do_IRQ+0x8a/0xb8
05/29/19 13:24:06 ?<00000000008e87f0>? ext_int_handler+0x134/0x138
05/29/19 13:24:06 ?<0000000000102cee>? enabled_wait+0x4e/0xe0
05/29/19 13:24:06 (?<0000000000001201>? 0x1201)
05/29/19 13:24:06 ?<000000000010303a>? arch_cpu_idle+0x32/0x48
05/29/19 13:24:06 ?<00000000001c5ae8>? do_idle+0xe8/0x1a8
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 01.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 04.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 05.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 00.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 06.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 02.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 03.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 virtual machine is placed in CP mode due to a SIGP stop from CPU 07.
05/29/19 13:24:06 ?<00000000001c5d86>? cpu_startup_entry+0x3e/0x48
05/29/19 13:24:06 ?<0000000000117240>? smp_start_secondary+0x120/0x140
05/29/19 13:24:06 ?<00000000008e8c46>? restart_int_handler+0x62/0x78
05/29/19 13:24:06 ?<0000000000000000>? (null)
05/29/19 13:24:06 Last Breaking-Event-Address:
05/29/19 13:24:06 ?<000000000012ba2e>? tlb_remove_table_rcu+0x4e/0x70
05/29/19 13:24:06
05/29/19 13:24:06 Kernel panic - not syncing: Fatal exception in interrupt
System Dump Location:
I will attach dumps here
*Additional Instructions for rjbrenn@us.ibm.com:
-Attach sysctl -a output output to the bug. |
|