Ubuntu 16.04.3 - call traces occurs when memory-hotplug test is run with 16Gb hugepages configured

[bugproxy]

Issue:

Call traces occurs when memory-hotplug script is run with 16Gb hugepages configured.

Environment:
ppc64le PowerVM Lpar

root@ltctuleta-lp1:~# uname -r
4.4.0-34-generic

root@ltctuleta-lp1:~# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 2
HugePages_Free: 2
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 16777216 kB

root@ltctuleta-lp1:~# free -h
              total used free shared buff/cache available
Mem: 85G 32G 52G 16M 193M 52G
Swap: 43G 0B 43G

Steps to reproduce:
1 - Download kernel source and enter to the directory- tools/testing/selftests/memory-hotplug/
2 - Run mem-on-off-test.sh script in it.

System gives call traces like:

offline_memory_expect_success 639: unexpected fail
online-offline 668
[ 57.552964] Unable to handle kernel paging request for data at address 0x00000028
[ 57.552977] Faulting instruction address: 0xc00000000029bc04
[ 57.552987] Oops: Kernel access of bad area, sig: 11 [#1]
[ 57.552992] SMP NR_CPUS=2048 NUMA pSeries
[ 57.553002] Modules linked in: btrfs xor raid6_pq pseries_rng sunrpc autofs4 ses enclosure nouveau bnx2x i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vxlan ip6_udp_tunnel ipr udp_tunnel rtc_generic mdio libcrc32c
[ 57.553050] CPU: 44 PID: 6518 Comm: mem-on-off-test Not tainted 4.4.0-34-generic #53-Ubuntu
[ 57.553059] task: c00000072773c8e0 ti: c000000727780000 task.ti: c000000727780000
[ 57.553067] NIP: c00000000029bc04 LR: c00000000029bbdc CTR: c0000000001107f0
[ 57.553076] REGS: c000000727783770 TRAP: 0300 Not tainted (4.4.0-34-generic)
[ 57.553083] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24242882 XER: 00000002
[ 57.553104] CFAR: c000000000008468 DAR: 0000000000000028 DSISR: 40000000 SOFTE: 1
GPR00: c00000000029bbdc c0000007277839f0 c0000000015b5d00 0000000000000000
GPR04: 000000000029d000 0000000000000800 0000000000000000 f00000000a000001
GPR08: f00000000a700020 0000000000000008 c00000000185e270 c000000e7e000050
GPR12: 0000000000002200 c00000000e6ea200 000000000029d000 0000000022000000
GPR16: 1000000000000000 c0000000015e2200 000000000a700000 0000000000000000
GPR20: 0000000000010000 0000000000000100 0000000000000200 c0000000015f16d0
GPR24: c000000001876510 0000000000000000 0000000000000001 c000000001872a00
GPR28: 000000000029d000 f000000000000000 f00000000a700000 000000000029c000
[ 57.553211] NIP [c00000000029bc04] dissolve_free_huge_pages+0x154/0x220
[ 57.553219] LR [c00000000029bbdc] dissolve_free_huge_pages+0x12c/0x220
[ 57.553226] Call Trace:
[ 57.553231] [c0000007277839f0] [c00000000029bbdc] dissolve_free_huge_pages+0x12c/0x220 (unreliable)
[ 57.553244] [c000000727783a80] [c0000000002dcbc8] __offline_pages.constprop.6+0x3f8/0x900
[ 57.553254] [c000000727783bd0] [c0000000006fbb38] memory_subsys_offline+0xa8/0x110
[ 57.553265] [c000000727783c00] [c0000000006d6424] device_offline+0x104/0x140
[ 57.553274] [c000000727783c40] [c0000000006fba80] store_mem_state+0x180/0x190
[ 57.553283] [c000000727783c80] [c0000000006d1e58] dev_attr_store+0x68/0xa0
[ 57.553293] [c000000727783cc0] [c000000000398110] sysfs_kf_write+0x80/0xb0
[ 57.553302] [c000000727783d00] [c000000000397028] kernfs_fop_write+0x188/0x200
[ 57.553312] [c000000727783d50] [c0000000002e190c] __vfs_write+0x6c/0xe0
[ 57.553321] [c000000727783d90] [c0000000002e2640] vfs_write+0xc0/0x230
[ 57.553329] [c000000727783de0] [c0000000002e367c] SyS_write+0x6c/0x110
[ 57.553339] [c000000727783e30] [c000000000009204] system_call+0x38/0xb4
[ 57.553346] Instruction dump:
[ 57.553351] 7e831836 4bfff991 e91e0028 e8fe0020 7d32e82a f9070008 f8e80000 fabe0020
[ 57.553366] fade0028 79294620 79291764 7d234a14 <e9030028> 3908ffff f9030028 81091458
[ 57.553383] ---[ end trace 617f7bdd75bcfc10 ]---
[ 57.557133]
Segmentation fault

The following commit IDs were built into a 4.10.0-37-generic #41 test kernel and verified to fix the problem:

a525108cf1cc14651602d678da38fa627a76a724
e1073d1e7920946ac4776a619cc40668b9e1401b
40692eb5eea209c2dd55857f44b4e1d7206e91d6
e24a1307ba1f99fc62a0bd61d5e87fcfb6d5503d
79cc38ded1e1ac86e69c90f604efadd50b0b3762
4ae279c2c96ab38a78b954d218790a8f6db714e5

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-10-17 06:53 EDT-------
Patches submitted to mailing list.

[Andrew Cloke (andrew-cloke)]

Are we waiting on the patches referenced in the bug description and comment #1 to land upstream before backporting?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-11-30 11:36 EDT-------
@Praveen,

any updates on your testing for this on a PowerNV system?

[Breno Leitão (breno-leitao)]

hi Andrew,

I understand that the patches are already accepted on upstream release, aren't they?

I can see them on Linus' tree.

[Andrew Cloke (andrew-cloke)]

Hi Breno, thanks for the update. I'll move the bug to "Triage" and ask the kernel team to take a look.

[Joseph Salisbury (jsalisbury)]

An Artful SRU request for these commits was already submitted in bug 1706247. I'm working on backporting them to Xenial(4.4 based), but have had some issues with build failures. There are some prereq commits that are also needed for 4.4 kernels. I'll see what the extent of these prereqs are and post an update.

If there are too many prereqs for Xenial, we may have to have this issues address in Xenial via the HWE kernels, but I'm investigating.

[Joseph Salisbury (jsalisbury)]

I built a Zesty test kernel with the six requested patches.

The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1706247/zesty

Can you test this kernel and see if it resolves this bug?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-12-19 05:45 EDT-------
Hi,

thank you for the patches. I'll test it.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-12-19 09:39 EDT-------
(In reply to comment #54)
> I built a Zesty test kernel with the six requested patches.
>
> The test kernel can be downloaded from:
>
> http://kernel.ubuntu.com/~jsalisbury/lp1706247/zesty
>
> Can you test this kernel and see if it resolves this bug?

Hi,

I wasn't able to find virtio.ko module in this package. Could you build an image with this module, please? Then, I can test this kernel on a LPAR machine as well.

Thank you

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-12-19 14:01 EDT-------
The bug is not reproducible in PowerNV with the kernel provided in comment #54.

Waiting for a package with virtio module to run tests on a VM.

[Joseph Salisbury (jsalisbury)]

Did you install both the linux-image and linux-image-extra .deb packages? I believer the virtio module in in -extra.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-12-20 08:19 EDT-------
Hi,

I did some tests on a LPAR machine. I got the following:

[root@tuletapio1-lp1 memory-hotplug]# sh ./mem-on-off-test.sh -r 80
[...]
[ 204.004656] Offlined Pages 4096
[ 204.008378] Offlined Pages 4096
[ 204.012030] Offlined Pages 4096
[ 204.012201] Built 2 zonelists in Node order, mobility grouping on. Total pages: 92347
[ 204.012210] Policy zone: DMA
[ 204.012283] ------------[ cut here ]------------
[ 204.012286] kernel BUG at /home/jsalisbury/bugs/lp1706247/zesty/ubuntu-zesty/mm/slub.c:3993!
[ 204.012291] Oops: Exception in kernel mode, sig: 5 [#1]
[ 204.012294] SMP NR_CPUS=2048
[ 204.012295] NUMA
[ 204.012297] pSeries
[ 204.012300] Modules linked in: xt_tcpudp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter btrfs xor raid6_pq vmx_crypto ip_tables x_tables xfs libcrc32c ibmveth ibmvscsi crc32c_vpmsum autofs4 [last unloaded: notifier_error_inject]
[ 204.012343] CPU: 18 PID: 6298 Comm: sh Not tainted 4.10.0-42-generic #46~lp1706247
[ 204.012347] task: c0000001f7dd8c00 task.stack: c0000001eef3c000
[ 204.012350] NIP: c000000000311134 LR: c000000000311148 CTR: c000000000310f40
[ 204.012353] REGS: c0000001eef3f6c0 TRAP: 0700 Not tainted (4.10.0-42-generic)
[ 204.012356] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
[ 204.012361] CR: 24222828 XER: 00000001
[ 204.012365] CFAR: c000000000311154 SOFTE: 1
GPR00: c000000000311148 c0000001eef3f940 c00000000145c900 0000000000000001
GPR04: c0000001fe010000 c0000001fe030000 c0000001fe030000 0000000000000000
GPR08: 0000000000000000 c0000001fe02fd88 0000000000000001 0000000000000000
GPR12: 0000000024222822 c00000000fb8a200 0000000000000bff f000000002ffffc0
GPR16: 0000000000000001 0000000000000000 c000000001483b00 0000000100001686
GPR20: 0000000000001000 0000000000000001 c0000001eef3fae0 c0000001ffec0600
GPR24: 0000000002fc0000 c0000000015ce368 f000000000000000 0000000000000000
GPR28: c000000001360738 0000000000000008 c000000001360758 c0000001fe02fd80
[ 204.012404] NIP [c000000000311134] slab_memory_callback+0x1f4/0x2b0
[ 204.012408] LR [c000000000311148] slab_memory_callback+0x208/0x2b0
[ 204.012410] Call Trace:
[ 204.012413] [c0000001eef3f940] [c000000000311148] slab_memory_callback+0x208/0x2b0 (unreliable)
[ 204.012419] [c0000001eef3f9a0] [c000000000112834] notifier_call_chain+0xa4/0x110
[ 204.012424] [c0000001eef3f9f0] [c000000000112d34] __blocking_notifier_call_chain+0x74/0xb0
[ 204.012429] [c0000001eef3fa40] [c000000000779a00] memory_notify+0x30/0x50
[ 204.012433] [c0000001eef3fa60] [c0000000003424cc] __offline_pages.constprop.8+0xa4c/0xa60
[ 204.012437] [c0000001eef3fbb0] [c000000000778afc] memory_block_action+0x9c/0x240
[ 204.012441] [c0000001eef3fc30] [c000000000779878] memory_subsys_offline+0x68/0xf0
[ 204...

[Joseph Salisbury (jsalisbury)]

Did the test kernel posted in comment #5 fix the original bug, but introduce a new issue?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-05 08:55 EDT-------
Laurent,

do you know maybe why this problem is occurring?

I get this problem always in the second run of the test script.

One think that is interesting is that if I reboot the system, it seems to work properly on the first run of the script, then, in the second run it fails.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-05 10:01 EDT-------
(In reply to comment #61)
> Laurent,
>
> do you know maybe why this problem is occurring?

I have no idea, but reading at the bug's history, it seems to be a new issue.

> I get this problem always in the second run of the test script.
>
> One think that is interesting is that if I reboot the system, it seems to
> work properly on the first run of the script, then, in the second run it
> fails.

Based on the stack trace, I would suggest to open a new bug for this issue.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-10 13:46 EDT-------
Hi Joseph,

so, it seems that we are dealing with a new bug!

Do you think it will be ok to open another bug with this kernel which you provided?

Thank you

[Joseph Salisbury (jsalisbury)]

Yes, it would be best to open a new bug. Just to confirm, this bug only happens when using the test kernel I posted in comment #5?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-16 10:45 EDT-------
Hi Joseph,

yes. We get a Segmentation Fault error without this patch an then I think we are unable to see that different error.

I'll open a new bug for it.

[Joseph Salisbury (jsalisbury)]

I removed the Zesty bug task since it is now EOL. Does this bug still happen with the latest HWE kernel for Xenial?

[Joseph Salisbury (jsalisbury)]

The latest HWE kernel should be 4.13 based.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-23 09:23 EDT-------
Hi Joseph,

I was able to reproduce the problem on HWE Kernal (XENIAL):

-- System

root@tuletapio2-lp3:~# uname -a
Linux tuletapio2-lp3 4.13.13 #1 SMP Tue Jan 23 07:41:39 CST 2018 ppc64le ppc64le ppc64le GNU/Linux

root@tuletapio2-lp3:~# cat /proc/meminfo
HugePages_Total: 2
HugePages_Free: 2
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 16777216 kB

root@tuletapio2-lp3:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.13.13 root=UUID=728bebfe-83ba-410d-917b-b552edbbb0a3 ro quiet splash default_hugepagesz=16G hugepagesz=16G hugepages=2

-- DUMP
Unable to handle kernel paging request for data at address 0x5deadbeef0000108
Faulting instruction address: 0xc0000000002cf374
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2048
NUMA
pSeries
Modules linked in:
CPU: 7 PID: 3769 Comm: mem-on-off-test Not tainted 4.13.13 #1
task: c0000007d9400000 task.stack: c0000007d9480000
NIP: c0000000002cf374 LR: c0000000002cf298 CTR: c000000000134ef0
REGS: c0000007d94837d0 TRAP: 0380 Not tainted (4.13.13)
MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
CR: 44422484 XER: 00000000
CFAR: c0000000002cf308 SOFTE: 1
GPR00: c0000000002cf28c c0000007d9483a50 c000000000e6ff00 c000000000fbc580
GPR04: f000000004000000 5deadbeef0000100 5deadbeef0000200 0000000000000002
GPR08: 5deadbeef0000000 0000000000000001 c000000000fbc580 c0000007fb000628
GPR12: 0000000000002200 c00000000fd02a00 0000000000001000 f000000004000000
GPR16: 0000000000000002 0000000000000000 c000000000ea3b00 0000000000001000
GPR20: 00000000ffffdb4f 0000000000000001 c000000fffd44600 c0000007d9483b60
GPR24: 0000000000100000 c000000000f0dad8 c000000000eb2dd0 0000000000000001
GPR28: 0000000000000000 c000000000fc4580 c000000000fd4580 f000000004000000
NIP [c0000000002cf374] dissolve_free_huge_page+0x124/0x230
LR [c0000000002cf298] dissolve_free_huge_page+0x48/0x230
Call Trace:
[c0000007d9483a50] [c0000000002cf28c] dissolve_free_huge_page+0x3c/0x230 (unreliable)
[c0000007d9483a90] [c0000000002cf548] dissolve_free_huge_pages+0xc8/0x150
[c0000007d9483ae0] [c0000000002ee1c8] __offline_pages.constprop.5+0x398/0xa90
[c0000007d9483c30] [c000000000645870] memory_subsys_offline+0x60/0xf0
[c0000007d9483c60] [c000000000623434] device_offline+0xf4/0x130
[c0000007d9483ca0] [c000000000645718] store_mem_state+0x178/0x190
[c0000007d9483ce0] [c00000000061ea34] dev_attr_store+0x34/0x60
[c0000007d9483d00] [c0000000003bdd10] sysfs_kf_write+0x60/0xa0
[c0000007d9483d20] [c0000000003bcaac] kernfs_fop_write+0x16c/0x240
[c0000007d9483d70] [c000000000314cf4] __vfs_write+0x34/0x70
[c0000007d9483d90] [c00000000031673c] vfs_write+0xcc/0x230
[c0000007d9483de0] [c000000000318410] SyS_write+0x60/0x110
[c0000007d9483e30] [c00000000000b860] system_call+0x58/0x6c
Instruction dump:
e90a0030 88ff0007 7fa94000 419e0120 3d005dea e8df0028 e8bf0020 7fe4fb78
6108dbee 7d435378 790807c6 6508f000 <f8c50008> f8a60000 7d094378 61080100
---[ end trace 23e3dda3fe0a58bd ]--

[Joseph Salisbury (jsalisbury)]

Can you see if this bug is resolved with the following artful kernel for bug 1706247:
http://kernel.ubuntu.com/~jsalisbury/lp1706247/artful

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-01-24 07:29 EDT-------
Hi,

The first problem seems to be not occurring anymore (the segmentation fault problem).

Although we are still facing the second problem (in the trace below), we created a new bug report for it.

root@tuletapio2-lp3:~/linux/tools/testing/selftests/memory-hotplug# uname -a
Linux tuletapio2-lp3 4.13.0-16-generic #19~lp1706247 SMP Wed Nov 1 23:21:53 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

[ 101.664761] kernel BUG at /home/jsalisbury/bugs/lp1706247/artful/ubuntu-artful/mm/slub.c:4034!
[ 101.664781] Oops: Exception in kernel mode, sig: 5 [#1]
[ 101.664785] SMP NR_CPUS=2048
[ 101.664785] NUMA
[ 101.664788] pSeries
[ 101.664792] Modules linked in: btrfs xor raid6_pq vmx_crypto crct10dif_vpmsum ip_tables x_tables autofs4 ibmveth ibmvscsi crc32c_vpmsum
[ 101.664807] CPU: 60 PID: 2151 Comm: sh Not tainted 4.13.0-16-generic #19~lp1706247
[ 101.664812] task: c000000fd2755900 task.stack: c000000fd27b8000
[ 101.664816] NIP: c0000000003630a4 LR: c0000000003630b8 CTR: c000000000150280
[ 101.664820] REGS: c000000fd27bb760 TRAP: 0700 Not tainted (4.13.0-16-generic)
[ 101.664824] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
[ 101.664829] CR: 24002828 XER: 00000000
[ 101.664835] CFAR: c0000000003630c4 SOFTE: 1
[ 101.664835] GPR00: c0000000003630b8 c000000fd27bb9e0 c000000001603300 0000000000000001
[ 101.664835] GPR04: c0000007fc010000 c0000007ffff3d80 f000000001ff03a0 0000000000000100
[ 101.664835] GPR08: c0000007ffff3c80 c0000007fc03fd90 0000000000000001 00000000ffffffff
[ 101.664835] GPR12: 0000000024002822 c00000000fae7600 f000000000000000 0000000000000000
[ 101.664835] GPR16: 00000000003fffff 0000000000140000 c0000000018e9400 f000000004ffffc0
[ 101.664835] GPR20: 0000000000000000 c000000001633b00 0000000000000002 00000000ffffb370
[ 101.664835] GPR24: 0000000000000001 0000000000001000 c000000fd27bbb70 0000000000000000
[ 101.664835] GPR28: c0000000014cadd8 0000000000000010 c0000000014cae38 c0000007fc03fd80
[ 101.664883] NIP [c0000000003630a4] slab_memory_callback+0x1d4/0x320
[ 101.664887] LR [c0000000003630b8] slab_memory_callback+0x1e8/0x320
[ 101.664890] Call Trace:
[ 101.664893] [c000000fd27bb9e0] [c0000000003630b8] slab_memory_callback+0x1e8/0x320 (unreliable)
[ 101.664901] [c000000fd27bba40] [c00000000012bacc] notifier_call_chain+0x9c/0x110
[ 101.664906] [c000000fd27bba90] [c00000000012c704] blocking_notifier_call_chain+0x64/0xa0
[ 101.664912] [c000000fd27bbad0] [c000000000824340] memory_notify+0x30/0x50
[ 101.664917] [c000000fd27bbaf0] [c000000000367678] __offline_pages.constprop.6+0xaa8/0xb10
[ 101.664923] [c000000fd27bbc30] [c0000000008241d8] memory_subsys_offline+0x68/0xd0
[ 101.664928] [c000000fd27bbc60] [c0000000007f7b68] device_offline+0xc8/0x140
[ 101.664933] [c000000fd27bbca0] [c000000000824040] store_mem_state+0x190/0x1b0
[ 101.664938] [c000000fd27bbce0] [c0000000007f27dc] dev_attr_store+0x3c/0x60
[ 101.664943] [c000000fd27bbd00] [c000000000469bd4] sysfs_kf_write+0x64/0x90
[ 101.664948] [c000000fd27bbd20] [c00000000046883c] kernfs_fop_write+0x1ac/0x270
[ 101.664954] [c000000fd27bbd70] [c00000000039d5cc] __vfs_write+0x3c...

[Joseph Salisbury (jsalisbury)]

Thanks for the update. I'll mark this bug as a duplicate of bug 1706247, since the fixes for that bug are the same for this bug.

We can look at the traces for the new bug in that bug report.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-18 09:56 EDT-------
Hi Joseph,

what is the status of this bug?

Tks

[Andrew Cloke (andrew-cloke)]

This bug has been marked as a duplicate of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1706247. As a result, no updates will be posted to this bug. Please goto LP#1706247 for the current status.
Thanks.