[Ubuntu 17.04] Kernel panics when large number of hugepages is passed as an boot argument to kernel.

[bugproxy]

Issue:
-----------
Kernel unable to handle paging request and panic occurs when more number of hugepages is passed as a boot argument to the kernel .

Environment:
----------------------
Power NV : Habanaro Bare metal
OS : Ubuntu 17.04
Kernel Version : 4.9.0-11-generic

Steps To reproduce:
-----------------------------------

1 - When the ubuntu Kernel boots try to add the boot argument 'hugepages = 12000000'.

The Kernel Panics and displays call traces like as below.

[ 5.030274] Unable to handle kernel paging request for data at address 0x00000000
[ 5.030323] Faulting instruction address: 0xc000000000302848
[ 5.030366] Oops: Kernel access of bad area, sig: 11 [#1]
[ 5.030399] SMP NR_CPUS=2048 [ 5.030416] NUMA
[ 5.039443] PowerNV
[ 5.039461] Modules linked in:
[ 5.050091] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.0-11-generic #12-Ubuntu
[ 5.053266] Workqueue: events pcpu_balance_workfn
[ 5.080647] task: c000003c8fe9b800 task.stack: c000003ffb118000
[ 5.090876] NIP: c000000000302848 LR: c0000000002709d4 CTR: c00000000016cef0
[ 5.094175] REGS: c000003ffb11b410 TRAP: 0300 Not tainted (4.9.0-11-generic)
[ 5.103040] MSR: 9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>[ 5.114466] CR: 22424222 XER: 00000000
[ 5.124932] CFAR: c000000000008a60 DAR: 0000000000000000 DSISR: 40000000 SOFTE: 1
GPR00: c0000000002709d4 c000003ffb11b690 c00000000141a400 c000003fff50e300
GPR04: 0000000000000000 00000000024001c2 c000003ffb11b780 000000219df50000
GPR08: 0000003ffb090000 c000000001454fd8 0000000000000000 0000000000000000
GPR12: 0000000000004400 c000000007b60000 00000000024001c2 00000000024001c2
GPR16: 00000000024001c2 0000000000000000 0000000000000000 0000000000000002
GPR20: 000000000000000c 0000000000000000 0000000000000000 00000000024200c0
GPR24: c0000000016eef48 0000000000000000 c000003fff50fd00 00000000024001c2
GPR28: 0000000000000000 c000003fff50fd00 c000003fff50e300 c000003ffb11b820
NIP [c000000000302848] mem_cgroup_soft_limit_reclaim+0xf8/0x4f0
[ 5.213613] LR [c0000000002709d4] do_try_to_free_pages+0x1b4/0x450
[ 5.230521] Call Trace:
[ 5.230643] [c000003ffb11b760] [c0000000002709d4] do_try_to_free_pages+0x1b4/0x450
[ 5.254184] [c000003ffb11b800] [c000000000270d68] try_to_free_pages+0xf8/0x270
[ 5.281896] [c000003ffb11b890] [c000000000259b88] __alloc_pages_nodemask+0x7a8/0xff0
[ 5.321407] [c000003ffb11bab0] [c000000000282cd0] pcpu_populate_chunk+0x110/0x520
[ 5.336262] [c000003ffb11bb50] [c0000000002841b8] pcpu_balance_workfn+0x758/0x960
[ 5.351526] [c000003ffb11bc50] [c0000000000ecdd0] process_one_work+0x2b0/0x5a0
[ 5.362561] [c000003ffb11bce0] [c0000000000ed168] worker_thread+0xa8/0x660
[ 5.374007] [c000003ffb11bd80] [c0000000000f5320] kthread+0x110/0x130
[ 5.385160] [c000003ffb11be30] [c00000000000c0e8] ret_from_kernel_thread+0x5c/0x74
[ 5.389456] Instruction dump:
[ 5.410036] eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 4e800020 3d230001 e9499a42 3d220004
[ 5.423598] 3929abd8 794a1f24 7d295214 eac90100 <e9360000> 2fa90000 419eff74 3b200000
[ 5.436503] ---[ end trace 23b650e96be5c549 ]---
[ 5.439700]

This is purely a negative scenario where the system does not have enough memory as the hugepages is given a very large argument.

Free output in a system:
free -h
              total used free shared buff/cache available
Mem: 251G 2.1G 248G 5.2M 502M 248G
Swap: 2.0G 159M 1.8G

The same scenario when tried after the linux is up like as,

echo 12000000 > /proc/sys/vm/nr_hugepages

HugePages_Total: 15069
HugePages_Free: 15069
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 16384 kB
root@ltc-haba2:~# free -h
              total used free shared buff/cache available
Mem: 251G 237G 13G 5.6M 311M 13G
Swap: 2.0G 159M 1.8G

In this case the kernel is able to allocate around 237 Gb for hugetlb.

But while the system is booting it gives us panic so please let know if this scenario is expected to be handled.

I identified the root cause of the panic.
When the system is running with low memory during mem cgroup initialisation, because most of the page have been grabbed to be huge pages, we hit a chicken and egg issue because when trying to allocate memory for the node's cgroup descriptor, we try to free some memory and in this path cgroup's services are called which assume node's cgroup descriptor is allocated.

I'm working on a patch which fixes this panic, but I think it is expected that the system fail due to OOM when all the pages are assigned to huge pages.

Patch sent upstream, waiting for review :
https://patchwork.kernel.org/patch/9573799/

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Michael Hohnbaum (hohnbaum)]

Leann,

Another one for the Kernel team to track.

                     Michael

On 02/15/2017 12:10 PM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> Issue:
> -----------
> Kernel unable to handle paging request and panic occurs when more number of hugepages is passed as a boot argument to the kernel .
>
> Environment:
> ----------------------
> Power NV : Habanaro Bare metal
> OS : Ubuntu 17.04
> Kernel Version : 4.9.0-11-generic
>
> Steps To reproduce:
> -----------------------------------
>
> 1 - When the ubuntu Kernel boots try to add the boot argument 'hugepages
> = 12000000'.
>
> The Kernel Panics and displays call traces like as below.
>
> [ 5.030274] Unable to handle kernel paging request for data at address 0x00000000
> [ 5.030323] Faulting instruction address: 0xc000000000302848
> [ 5.030366] Oops: Kernel access of bad area, sig: 11 [#1]
> [ 5.030399] SMP NR_CPUS=2048 [ 5.030416] NUMA
> [ 5.039443] PowerNV
> [ 5.039461] Modules linked in:
> [ 5.050091] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.0-11-generic #12-Ubuntu
> [ 5.053266] Workqueue: events pcpu_balance_workfn
> [ 5.080647] task: c000003c8fe9b800 task.stack: c000003ffb118000
> [ 5.090876] NIP: c000000000302848 LR: c0000000002709d4 CTR: c00000000016cef0
> [ 5.094175] REGS: c000003ffb11b410 TRAP: 0300 Not tainted (4.9.0-11-generic)
> [ 5.103040] MSR: 9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>[ 5.114466] CR: 22424222 XER: 00000000
> [ 5.124932] CFAR: c000000000008a60 DAR: 0000000000000000 DSISR: 40000000 SOFTE: 1
> GPR00: c0000000002709d4 c000003ffb11b690 c00000000141a400 c000003fff50e300
> GPR04: 0000000000000000 00000000024001c2 c000003ffb11b780 000000219df50000
> GPR08: 0000003ffb090000 c000000001454fd8 0000000000000000 0000000000000000
> GPR12: 0000000000004400 c000000007b60000 00000000024001c2 00000000024001c2
> GPR16: 00000000024001c2 0000000000000000 0000000000000000 0000000000000002
> GPR20: 000000000000000c 0000000000000000 0000000000000000 00000000024200c0
> GPR24: c0000000016eef48 0000000000000000 c000003fff50fd00 00000000024001c2
> GPR28: 0000000000000000 c000003fff50fd00 c000003fff50e300 c000003ffb11b820
> NIP [c000000000302848] mem_cgroup_soft_limit_reclaim+0xf8/0x4f0
> [ 5.213613] LR [c0000000002709d4] do_try_to_free_pages+0x1b4/0x450
> [ 5.230521] Call Trace:
> [ 5.230643] [c000003ffb11b760] [c0000000002709d4] do_try_to_free_pages+0x1b4/0x450
> [ 5.254184] [c000003ffb11b800] [c000000000270d68] try_to_free_pages+0xf8/0x270
> [ 5.281896] [c000003ffb11b890] [c000000000259b88] __alloc_pages_nodemask+0x7a8/0xff0
> [ 5.321407] [c000003ffb11bab0] [c000000000282cd0] pcpu_populate_chunk+0x110/0x520
> [ 5.336262] [c000003ffb11bb50] [c0000000002841b8] pcpu_balance_workfn+0x758/0x960
> [ 5.351526] [c000003ffb11bc50] [c0000000000ecdd0] process_one_work+0x2b0/0x5a0
> [ 5.362561] [c000003ffb11bce0] [c0000000000ed168] worker_thread+0xa8/0x660
> [ 5.374007] [c000003ffb11bd80] [c0000000000f5320] kthread+0x110/0x130
> [ 5.385160] [c000003ffb11be30] [c00000000000c0e8] ret_from_kernel_thread+0x5c/0x74
> [ 5.389456] Instruction...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-02-21 02:51 EDT-------
As discussed with Michal Hocko, I will write a larger patch which is delayed the soft limit's data as it is rarely used. This will extend the patch I already sent.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-02-23 08:51 EDT-------
A new set of 2 patches have been sent to the community :
https://patchwork.kernel.org/patch/9588337/
https://patchwork.kernel.org/patch/9588335/

I'm waiting for these 2 patches to be accepted upstream.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-03-02 05:37 EDT-------
Based on the discussion in the mm mailing list, only the first patch is going to be accepted upstream:
https://patchwork.kernel.org/patch/9588337/

[Tim Gardner (timg-tpi)]

Applied 'mm/cgroup: avoid panic when init with low memory' to the Zesty 4.10 kernel

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.10.0-13.15

---------------
linux (4.10.0-13.15) zesty; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1671614

  * ehci-platform needed in usb-modules udeb (LP: #1671589)
    - d-i: add ehci-platform to usb-modules

  * irqchip/gic-v3-its: Enable cacheable attribute Read-allocate hints
    (LP: #1671598)
    - irqchip/gic-v3-its: Enable cacheable attribute Read-allocate hints

  * iommu: Fix static checker warning in iommu_insert_device_resv_regions
    (LP: #1671599)
    - iommu: Fix static checker warning in iommu_insert_device_resv_regions

  * QDF2400: Fix panic introduced by erratum 1003 (LP: #1671602)
    - arm64: Avoid clobbering mm in erratum workaround on QDF2400

  * QDF2400 PCI ports require ACS quirk (LP: #1671601)
    - PCI: Add ACS quirk for Qualcomm QDF2400 and QDF2432

  * tty: pl011: Work around QDF2400 E44 stuck BUSY bit (LP: #1671600)
    - tty: pl011: Work around QDF2400 E44 stuck BUSY bit

  * CVE-2017-2636
    - tty: n_hdlc: get rid of racy n_hdlc.tbuf

  * Sync virtualbox to 5.1.16-dfsg-1 in zesty (LP: #1671470)
    - ubuntu: vbox -- Update to 5.1.16-dfsg-1

 -- Tim Gardner <email address hidden> Thu, 09 Mar 2017 06:16:24 -0700

[bugproxy (bugproxy)]

Default Comment by Bridge