LSK: BUG: failure at change_huge_pmd in huge_memory.c

[Naresh Kamboju]

While testing LTP test suite and memory stress tests the following bug found.

Test case name: thp03

Error log:
--------------
 thp02 0 TINFO : mremap 0x7f90e01000 to 0x7f8fa01000
 thp02 1 TPASS : Still alive.
 [ 2525.441528] BUG: failure at /home/buildslave/workspace/linux-linaro-stable-lsk-pre-merge/mm/huge_memory.c:1490/change_huge_pmd()!
 [ 2525.455612] Kernel panic - not syncing: BUG!
 [ 2525.459900] CPU: 5 PID: 5959 Comm: thp03 Not tainted 3.10.39+ #1
 [ 2525.465924] Call trace:
 [ 2525.468388] [<ffffffc000087464>] dump_backtrace+0x0/0x12c
 [ 2525.473807] [<ffffffc0000875a0>] show_stack+0x10/0x1c
 [ 2525.478879] [<ffffffc0005f36c4>] dump_stack+0x1c/0x28
 [ 2525.483950] [<ffffffc0005ef0f4>] panic+0xe4/0x208
 [ 2525.488673] [<ffffffc00016fffc>] change_huge_pmd+0xf0/0xf4
 [ 2525.494178] [<ffffffc000156d7c>] change_protection+0x39c/0x49c
 [ 2525.500031] [<ffffffc000156f5c>] mprotect_fixup+0xe0/0x204
 [ 2525.505534] [<ffffffc000157180>] SyS_mprotect+0x100/0x210
 [ 2525.510953] CPU0: stopping
 [ 2525.513677] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.39+ #1
 [ 2525.519787] Call trace:
 [ 2525.522249] [<ffffffc000087464>] dump_backtrace+0x0/0x12c
 [ 2525.527667] [<ffffffc0000875a0>] show_stack+0x10/0x1c
 [ 2525.532737] [<ffffffc0005f36c4>] dump_stack+0x1c/0x28
 [ 2525.537808] [<ffffffc00008dea0>] handle_IPI+0x15c/0x174
 [ 2525.543050] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
 [ 2525.548464] Exception stack(0xffffffc000827df0 to 0xffffffc000827f10)
...
Complete Error log attached to this bug and same can be found in this link.
https://validation.linaro.org/scheduler/job/127011/log_file#L_27_9641

Bug reported kernel code snippet
-------------------------------------------------
File: mm/huge_memory.c

1479 int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
1480 unsigned long addr, pgprot_t newprot, int prot_numa)
1481 {
1482 struct mm_struct *mm = vma->vm_mm;
1483 int ret = 0;
1484
1485 if (__pmd_trans_huge_lock(pmd, vma) == 1) {
1486 pmd_t entry;
1487 entry = pmdp_get_and_clear(mm, addr, pmd);
1488 if (!prot_numa) {
1489 entry = pmd_modify(entry, newprot);
1490 BUG_ON(pmd_write(entry));

Test case description:
Test case link: http://people.linaro.org/~naresh.kamboju/ltp/testcases/kernel/mem/thp/thp03.c

/*
 * thp03 - Case for spliting unaligned memory.
 * - System will panic if failed.
 *
 * Modified form a reproducer for
 * https://patchwork.kernel.org/patch/1358441/
 * Kernel Commit id: 027ef6c87853b0a9df53175063028edb4950d476
 * There was a bug in THP, will crash happened due to the following
 * reason according to developers:
 *
 * most VM places are using pmd_none but a few are still using
 * pmd_present. The meaning is about the same for the pmd. However
 * pmd_present would return the wrong value on PROT_NONE ranges or in
 * case of a non reproducible race with split_huge_page.
 * When the code using pmd_present gets a false negative, the kernel will
 * crash. It's just an annoying DoS with a BUG_ON triggering: no memory
 * corruption and no data corruption (nor userland nor kernel).
 */

Linux kernel version:
---------------------
Linux version 3.10.39+ (buildslave@x86-64-07) (gcc version 4.8.3 20140203 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2014.02 - Linaro GCC 2014.02) ) #1 SMP Fri May 16 08:37:37 UTC 2014
CPU: AArch64 Processor [410fd030] revision 0
Machine: Juno

Please let know if you need more information regarding this bug

[Steve Capper (steve-capper)]

Hi,
Could you please merge the following:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=44b6dfc556811c0e6981329e29e1865fc7a8847d

and,
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c2c93e5b7f3f42277ec25ff3746096abc0c0d0f7

Thanks,
--
Steve

[Steve Capper (steve-capper)]

Also, could you please merge this one:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9c7e535fcc1725fc2e2d4f0d9dd14137f0243e23

Cheers,
--
Steve

[Milosz Wasilewski (mwasilew)]

Same bug when using 32bit LTP on 64bit kernel:
https://validation.linaro.org/scheduler/job/127055/log_file#L_29_9531

[Mark Brown (broonie)]

Steve, please send followups for LSK bugs via e-mail (commenting on the bug is fine too) - unfortunately due to some hilarity with the way the LAVA lab is set up I get copies of all Launchpad bugmail for LAVA which gmail isn't capable of filtering out.

Milosz, any chance we could get the 32 bit CI set up?

[Mark Brown (broonie)]

Steve, the first commit you mention above (arm64: mm: Remove PTE_BIT_FUNC macro) is already present as is the second (arm64: mm: Introduce PTE_WRITE). Is there something else going on here?

[Milosz Wasilewski (mwasilew)]

@broonie
do you mean 64bit kernel + 32bit rootfs CI?

[Mark Brown (broonie)]

@mwasilew: Yes, please.

[Steve Capper (steve-capper)]

Hi Mark,
What about:
9c7e535fcc1725fc2e2d4f0d9dd14137f0243e23
"arm64: mm: Route pmd thp functions through pte equivalents"

If that doesn't work (or is already in), can you please send me the
git branch corresponding to the failure and I'll look at it when I'm
back from holiday (3rd June).

Cheers,
--
Steve

[Mark Brown (broonie)]

That's in there. There's a few bits and pieces I'm still catching up on but the git branch is the standard LSK - git://git.linaro.org/kernel/linux-linaro-stable.git linux-linaro-lsk

[Steve Capper (steve-capper)]

A bug has been found by Will in the way pmds are set in THP.

The following patch fixes some problems Will experienced in LTP:
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-May/260476.html
This is in 3.15-rc8 as:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ceb218359de22e70980801d4fa04fffbfc44adb8

I see this is already in LSK, has this fixed the issue?

Cheers,
--
Steve

On 27 May 2014 16:27, Mark Brown <email address hidden> wrote:
> That's in there. There's a few bits and pieces I'm still catching up on
> but the git branch is the standard LSK - git://git.linaro.org/kernel
> /linux-linaro-stable.git linux-linaro-lsk
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1320782
>
> Title:
> LSK: BUG: failure at change_huge_pmd in huge_memory.c
>
> Status in Linaro Stable Kernel (LSK):
> New
>
> Bug description:
> While testing LTP test suite and memory stress tests the following bug
> found.
>
> Test case name: thp03
>
> Error log:
> --------------
> thp02 0 TINFO : mremap 0x7f90e01000 to 0x7f8fa01000
> thp02 1 TPASS : Still alive.
> [ 2525.441528] BUG: failure at /home/buildslave/workspace/linux-linaro-stable-lsk-pre-merge/mm/huge_memory.c:1490/change_huge_pmd()!
> [ 2525.455612] Kernel panic - not syncing: BUG!
> [ 2525.459900] CPU: 5 PID: 5959 Comm: thp03 Not tainted 3.10.39+ #1
> [ 2525.465924] Call trace:
> [ 2525.468388] [<ffffffc000087464>] dump_backtrace+0x0/0x12c
> [ 2525.473807] [<ffffffc0000875a0>] show_stack+0x10/0x1c
> [ 2525.478879] [<ffffffc0005f36c4>] dump_stack+0x1c/0x28
> [ 2525.483950] [<ffffffc0005ef0f4>] panic+0xe4/0x208
> [ 2525.488673] [<ffffffc00016fffc>] change_huge_pmd+0xf0/0xf4
> [ 2525.494178] [<ffffffc000156d7c>] change_protection+0x39c/0x49c
> [ 2525.500031] [<ffffffc000156f5c>] mprotect_fixup+0xe0/0x204
> [ 2525.505534] [<ffffffc000157180>] SyS_mprotect+0x100/0x210
> [ 2525.510953] CPU0: stopping
> [ 2525.513677] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.39+ #1
> [ 2525.519787] Call trace:
> [ 2525.522249] [<ffffffc000087464>] dump_backtrace+0x0/0x12c
> [ 2525.527667] [<ffffffc0000875a0>] show_stack+0x10/0x1c
> [ 2525.532737] [<ffffffc0005f36c4>] dump_stack+0x1c/0x28
> [ 2525.537808] [<ffffffc00008dea0>] handle_IPI+0x15c/0x174
> [ 2525.543050] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
> [ 2525.548464] Exception stack(0xffffffc000827df0 to 0xffffffc000827f10)
> ...
> Complete Error log attached to this bug and same can be found in this link.
> https://validation.linaro.org/scheduler/job/127011/log_file#L_27_9641
>
> Bug reported kernel code snippet
> -------------------------------------------------
> File: mm/huge_memory.c
>
> 1479 int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
> 1480 unsigned long addr, pgprot_t newprot, int prot_numa)
> 1481 {
> 1482 struct mm_struct *mm = vma->vm_mm;
> 1483 int ret = 0;
> 1484
> 1485 if (__pmd_trans_huge_lock(pmd, vma) == 1) {
> 1486 pmd_t entry;
> 1487 entry ...

[Mark Brown (broonie)]

Not visibly but the CI jobs seem to have stopped running unless I kick them by hand - I just did that, you should be able to see the results in LAVA soonish.

[Mark Brown (broonie)]

Didn't seem to have any impact.

[Steve Capper (steve-capper)]

Could you please revert:

2d89dd0 arm64: Introduce execute-only page access permissions

And give it another go? This appears to be skipping the write logic
and causing the problems to manifest in THP03.

cc: Catalin for info.

Cheers,
--
Steve

[Catalin Marinas (catalin-marinas)]

On Wed, Jun 04, 2014 at 09:34:06AM +0100, Steve Capper wrote:
> Could you please revert:
>
> 2d89dd0 arm64: Introduce execute-only page access permissions

Do you mean commit "arm64: Introduce execute-only page access
permissions" is in LSK? It's been in linux-next temporarily and I
reverted it in the arm64/for-next/core tree (and linux-next) for lack of
more intensive testing. But it really should not have been picked into
LSK since it's not even in mainline.

Thanks.

--
Catalin

[Mark Brown (broonie)]

Unfortunately due to the strong demand from users to get features in we're pretty much being forced to track -next rather than Linus' tree (as well as things like Lorenzo's cpuidle PSCI integration that haven't made it in).

I will have got around to picking up the revert if it's done as a revert (which it is) anyway as part of my regular catchups, I'll go pull this one a little sooner and see if that helps.

[Mark Brown (broonie)]

That revert doesn't seem to have made much difference unfortunately.

[Steve Capper (steve-capper)]

Okay,
So going through the logs, and testing on my board. The
change_huge_pmd problem has been solved.
There are some problems with libhugetlbfs unit tests, as they test fixes > 3.10.
I managed to get all the tests passing by cherry-picking:
07443a8 mm, hugetlb: return a reserved page to a reserved pool if failed
8312034 mm, hugetlb: grab a page_table_lock after page_cache_release
5944d01 mm, hugetlb: remove useless check about mapping type
8bb3f12 mm, hugetlb: fix subpool accounting handling
f522c3a mm, hugetlb: change variable name reservations to resv
4ef9184 mm, hugetlb: protect reserved pages when soft offlining a hugepage
af0ed73 mm, hugetlb: decrement reserve count if VM_NORESERVE alloc page cache
a63884e mm, hugetlb: remove decrement_hugepage_resv_vma()
72231b0 mm, hugetlb: add VM_NORESERVE check in vma_has_reserves()
37a2140 mm, hugetlb: do not use a page in page cache for cow optimization
c0d934b mm, hugetlb: remove redundant list_empty check in gather_surplus_pages()
b226102 mm, hugetlb: fix and clean-up node iteration code to alloc or free
81a6fca mm, hugetlb: clean-up alloc_huge_page()

This will require some thorough testing.

Cheers,
--
Steve

[Mark Brown (broonie)]

Do you have your cherry picks somewhere I can pull?

[Steve Capper (steve-capper)]

Hi Mark,
I've put them up here:
https://git.linaro.org/people/steve.capper/linux.git/shortlog/refs/heads/for-broonie/lsk-hugetlb

Cheers,
--
Steve

[Mark Brown (broonie)]

Great, thanks.

[Mark Brown (broonie)]

Ah, I see you applied everything in the opposite order to that listed - that makes things much easier! I've applied everything and thrown it at CI (should be pre-merge build 86), let's see what happens.

One thing if you're applying other people's commits: please remember to always add a Signed-off-by: - it's important for legal reasons. git cherry-pick and am both have a "--signoff" option for this.

[Mark Brown (broonie)]

Initial results on hardware show that it's made things happier - we now have more tests passing, though there is still one failure left. Only just saw this, not taken a look at all:

https://validation.linaro.org/dashboard/streams/private/team/linaro/ci-linux-linaro-stable-lsk-pre-merge/bundles/fb2729d88e75cd4153db4e529ee47f3f223245ef/

Still waiting for the models but I expect they are similar, most likely even better (they had one more test passing previous to this update). The still failing test is icache-hygene which has been passing on the models and failing on the hardware.

[Mark Brown (broonie)]

(Assuming the models are fine and other CI looks good I'll probably push these updates since they appear to be a substantial win all round.)

[Mark Brown (broonie)]

Yup, 90/90 on the model - thanks Steve!