memory_stress_ng failing for Power architecture for 16.04

[Mike Rushton]

memory_stress_ng, as part of server certification is failing for IBM Power S812LC(TN71-BP012) in bare metal mode. Failing in this case is defined by the test locking up the server in an unrecoverable state which only a reboot will fix.

I will be attaching screen and kern logs for the failures and a successful run on 14.04 on the same server.

[Mike Rushton (leftyfb)]

screen session of failure

[Mike Rushton (leftyfb)]

kern.log from failure

[Mike Rushton (leftyfb)]

screen session log from successful test on 14.04

[Mike Rushton (leftyfb)]

kern.log from successful test on 14.04

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1573062

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Colin Ian King (colin-king)]

I don't see any evidence of a hang, just evidence of a machine being power-cycled.

Apr 19 21:26:33 ubuntu kernel: [19749.994340] [45742] 1000 45742 369 18 6 3 14 0 stress-ng-brk
Apr 19 21:26:33 ubuntu kernel: [19749.994342] [45743] 1000 45743 369 18 6 3 14 0 stress-ng-brk
Apr 19 21:26:33 ubuntu kernel: [19749.994344] Out of memory: Kill process 45583 (stress-ng-brk) score 28 or sacrifice child
Apr 19 21:26:33 ubuntu kernel: [19749.994566] Killed process 45583 (stress-ng-brk) total-vm:7976960kB, anon-rss:7048512kB, file-rss:1152kB
Apr 20 14:28:33 binacle kernel: [ 0.000000] opal: OPAL V3 detected !
Apr 20 14:28:33 binacle kernel: [ 0.000000] Allocated 4980736 bytes for 2048 pacas at c00000000fb40000
Apr 20 14:28:33 binacle kernel: [ 0.000000] Using PowerNV machine description
Apr 20 14:28:33 binacle kernel: [ 0.000000] Page sizes from device-tree:
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
Apr 20 14:28:33 binacle kernel: [ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1

Can you supply details of how stress-ng is being run?

For this brk stressor test, stress-ng performs rapid heap expansion using the brk() system call, which will force the system to consume all available memory and then transition into a swapping phase. This can lead to an apparent "hung" situation (for example, your shell may be swapped out), even though it is still active and not crashed.

The kernel messages you see in the kern.log are just the kernel OOM killer killing off the best candidate process to free up memory; this free'd memory will be consumed rapidly again by other contending brk stressors so the system will appear to be hung while it is busy cycling around on the killing and spawning of these stressors.

Was the machine ping'able? If so, it's not dead/hung. Perhaps it got power cycled prematurely. What evidence was there that the machine was hung?

[Mike Rushton (leftyfb)]

The very first line of the screen session shows how the test is being run:

/usr/lib/plainbox-provider-checkbox/bin/memory_stress_ng

The machine is completely unresponsive. No ssh, no ping and the console over ipmi is locked up. Hitting CTRL+C over the sol console does nothing at all. This is after leaving the test running for 18+ hours where it should only take around 1-3 hours tops to complete depending on the machine. As you can see on the same machine running the same test on 14.04, the test took about an hour to complete with no lockup at the end.

[Rod Smith (rodsmith)]

Note that memory_stress_ng is a script in the Checkbox certification suite. The actual stress-ng command line should be:

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

The exact values of $end_time and $runtime vary with the amount of memory in the system -- it's 300 seconds plus 10 seconds per GiB for $runtime and 50% more for $end_time.

Note also that Mike's "1-3 hours tops" refers to the ENTIRE memory_stress_ng run; the "stress-ng... --brk 0" run would of course be much shorter than that, since the script runs a series of stressors in sequence.

[Colin Ian King (colin-king)]

Is this being run as a normal user or with root privileges?

[Rod Smith (rodsmith)]

It should be run as root.

[Colin Ian King (colin-king)]

Ahah, perhaps one should examine the stress-ng manual:

"Running stress-ng with root privileges will adjust out of memory settings on Linux systems to make the stressors unkillable in low memory situations, so use this judiciously. "

We may be just being a bit too demanding...

[Mike Rushton (leftyfb)]

The tests run manually above were NOT run as root. The test during certification does run as root. I have run the test as root and as the ubuntu user, both with the same results.

Also, as shown above, this test ran fine manually while running Ubuntu 14.04 without root and during proper certification testing as root.

[Joseph Salisbury (jsalisbury)]

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

3.16: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.16-utopic/
3.19: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/
4.2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2-wily/

You don't have to test every kernel, just up until the kernel that first has this bug.

Once we know the results of those kernels, we can narrow it down further.

Thanks in advance!

[Mike Rushton (leftyfb)]

Are there ppc64le versions of those kernels?

[Tim Gardner (timg-tpi)]

Mike - Wily was the first release to support ppc64el - http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2-wily/linux-image-4.2.0-040200-generic_4.2.0-040200.201510260713_ppc64el.deb

[Mike Rushton (leftyfb)]

Confirmed the above kernel fails in the same manner.

[Joseph Salisbury (jsalisbury)]

Can you test the 4.1 kernel next? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-wily/

We may also want to test the latest mainline kernel to see if this bug is already fixed there. If it is, we can perform a "Reverse" bisect to find the fix. The mainline kernel is available from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc6-wily/

[Mike Rushton (leftyfb)]

Ubuntu 14.04
3.19.0-58-generic - pass
4.2.0-35-generic - fails
4.1.0-040100-generic - fail
4.6.0-040600rc6-generic - pass

Ubuntu 16.04
4.4.0-22-generic - fail
4.1.0-040100-generic - fail
4.6.0-040600rc6-generic - pass

[Joseph Salisbury (jsalisbury)]

That is good news that the bug is fixed in mainline. We can perform a reverse bisect and find the commit that fixes this. We first need to narrow down which version fixes the issue. Can you test the following kernels next:

v4.5: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5-wily/
v4.6-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc1-wily/

[Colin Ian King (colin-king)]

I'd like to chip in with my $0.02 worth. This may be a sporadic hang, so it may be worth double checking each bisection point in case it's a bug that may be a little more racy than we'd like.

[Mike Rushton (leftyfb)]

I have noticed only on the Alpine hardware within an LPAR that is has been sporadic. I assume that is due to the additional hardware resources that can better handle the memory/stress test. All other tests have been 2 or 3 times each. More more than that. I can run the above kernel tests, but mind you, each test takes several hours to half a days worth. Running each test multiple times could take about a week to complete. We really need to get this sorted out ASAP since it is holding up all Power certification.

[Mike Rushton (leftyfb)]

4.5.0-040500.201603140130_ppc64el - fail
4.6.0-040600rc1.201603261930_ppc64el - pass

4.6 was run twice and passed both times.

[Joseph Salisbury (jsalisbury)]

I started a "Reverse" kernel bisect between v4.5 final and v4.6-rc1. The kernel bisect will require testing of about 10-12 test kernels.

I built the first test kernel, up to the following commit:

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

In parallel to the bisect, I'll review the git logs to see if the fix sticks out. There are 166 powerpc commits between these two versions, so a bisect may be the only way to identify the exact commit.

Thanks in advance

[Mike Rushton (leftyfb)]

Testing now...

[Mike Rushton (leftyfb)]

So far, after 2 runs, 4.5.0-040500-generic_4.5.0-040500.201605161244 seems to be working. I'm running it a 3rd time now just to be sure. It takes about 7 hours to finish.

[Joseph Salisbury (jsalisbury)]

Thanks for the update, Mike. If you think the two test runs are enough to prove that kernel is good, I'll build the next one. Otherwise, I'll wait for the results of the third run.

[Mike Rushton (leftyfb)]

I would go ahead and assume it's good. I haven't had 2 tests run good twice in a row yet.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
53d2e6976bd4042672ed7b90dfbf4b31635b7dcf

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Joseph Salisbury (jsalisbury)]

Commit 6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f was actually bad. So the test kernel in comment #28 is invalid. I'll re-update the bisect and build the next kernel.

[Joseph Salisbury (jsalisbury)]

I reset the bisect and built the first test kernel, up to the following commit:
96b9b1c95660d4bc5510c5d798d3817ae9f0b391

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Joseph Salisbury (jsalisbury)]

Sorry we should actually test the kernel in comment #28. This is because we are performing a "Reverse" bisect and not a regular bisect. The test kernel is available from:

http://kernel.ubuntu.com/~jsalisbury/lp1573062

The .deb file name is:
linux-image-4.5.0-040500-generic_4.5.0-040500.201605171117_ppc64el.deb

Thanks, and sorry for the confusion.

[Mike Rushton (leftyfb)]

4.5.0-040500.201605171117 failed

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
2e11590171683c6b12193fe4b0ede1e6201b7f45

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

PASS,2016-05-23-13-21-37,4.5.0-040500-generic-201605230752
FAIL,2016-05-23-21-06-07,4.5.0-040500-generic-201605230752
PASS,2016-05-24-04-32-39,4.5.0-040500-generic-201605230752
FAIL,2016-05-24-13-31-01,4.5.0-040500-generic-201605230752
FAIL,2016-05-25-04-06-51,4.5.0-040500-generic-201605230752

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
5a010c73cdb760c9bdf37b28824b6566789cc005

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

4.5.0-040500-generic-201605251558 failed

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
8f40842e4260f73792c156aded004197a19135ee

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-05-27 03:40 EDT-------
Hi, where could I get the src/binary of memory_stress_ng. I will try to reproduce it in local power servers

[Michael Ellerman (michael-ellerman)]

Can you please post the bisection log using upstream commit ids.

[Mike Rushton (leftyfb)]

8f40842e4260f73792c156aded004197a19135ee failed

[Joseph Salisbury (jsalisbury)]

Here is the current bisect log(This is a "Reverse" bisect so good means bad and vice versa):

git bisect start
# good: [b562e44f507e863c6792946e4e1b1449fbbac85d] Linux 4.5
git bisect good b562e44f507e863c6792946e4e1b1449fbbac85d
# bad: [f55532a0c0b8bb6148f4e07853b876ef73bc69ca] Linux 4.6-rc1
git bisect bad f55532a0c0b8bb6148f4e07853b876ef73bc69ca
# good: [6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f] Merge branch 'for-4.6' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect good 6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f
# good: [53d2e6976bd4042672ed7b90dfbf4b31635b7dcf] Merge tag 'xfs-for-linus-4.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs
git bisect good 53d2e6976bd4042672ed7b90dfbf4b31635b7dcf
# good: [2e11590171683c6b12193fe4b0ede1e6201b7f45] staging: fsl-mc: fix incorrect type passed to dev_err macros
git bisect good 2e11590171683c6b12193fe4b0ede1e6201b7f45
# good: [5a010c73cdb760c9bdf37b28824b6566789cc005] Merge tag 'platform-drivers-x86-v4.6-1' of git://git.infradead.org/users/dvhart/linux-platform-drivers-x86
git bisect good 5a010c73cdb760c9bdf37b28824b6566789cc005
# good: [8f40842e4260f73792c156aded004197a19135ee] Merge tag 'for-linus-20160324' of git://git.infradead.org/linux-mtd
git bisect good 8f40842e4260f73792c156aded004197a19135ee

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
45996492e5c85aa0ac93a95d1b2d1ed56851c865

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

45996492e5c85aa0ac93a95d1b2d1ed56851c865 failed

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
d9dddbf556674bf125ecd925b24e43a5cf2a568a

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

d9dddbf556674bf125ecd925b24e43a5cf2a568a failed

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
895a1067d5b83065afbad3bb02c3c464b71f1b3f

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

895a1067d5b83065afbad3bb02c3c464b71f1b3f failed

[Michael Ellerman (michael-ellerman)]

Can you post the latest bisect log?

[Joseph Salisbury (jsalisbury)]

Commit failed:
89f081730c49a1d3b46359aa0054e6b3b80f47e4

[Joseph Salisbury (jsalisbury)]

Current "REVERSE" bisect log:

git bisect start
# good: [b562e44f507e863c6792946e4e1b1449fbbac85d] Linux 4.5
git bisect good b562e44f507e863c6792946e4e1b1449fbbac85d
# bad: [f55532a0c0b8bb6148f4e07853b876ef73bc69ca] Linux 4.6-rc1
git bisect bad f55532a0c0b8bb6148f4e07853b876ef73bc69ca
# good: [6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f] Merge branch 'for-4.6' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect good 6b5f04b6cf8ebab9a65d9c0026c650bb2538fd0f
# good: [53d2e6976bd4042672ed7b90dfbf4b31635b7dcf] Merge tag 'xfs-for-linus-4.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs
git bisect good 53d2e6976bd4042672ed7b90dfbf4b31635b7dcf
# good: [2e11590171683c6b12193fe4b0ede1e6201b7f45] staging: fsl-mc: fix incorrect type passed to dev_err macros
git bisect good 2e11590171683c6b12193fe4b0ede1e6201b7f45
# good: [5a010c73cdb760c9bdf37b28824b6566789cc005] Merge tag 'platform-drivers-x86-v4.6-1' of git://git.infradead.org/users/dvhart/linux-platform-drivers-x86
git bisect good 5a010c73cdb760c9bdf37b28824b6566789cc005
# good: [8f40842e4260f73792c156aded004197a19135ee] Merge tag 'for-linus-20160324' of git://git.infradead.org/linux-mtd
git bisect good 8f40842e4260f73792c156aded004197a19135ee
# good: [45996492e5c85aa0ac93a95d1b2d1ed56851c865] orangefs: fix orangefs_superblock locking
git bisect good 45996492e5c85aa0ac93a95d1b2d1ed56851c865
# good: [d9dddbf556674bf125ecd925b24e43a5cf2a568a] mm/page_alloc: prevent merging between isolated and other pageblocks
git bisect good d9dddbf556674bf125ecd925b24e43a5cf2a568a
# good: [895a1067d5b83065afbad3bb02c3c464b71f1b3f] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
git bisect good 895a1067d5b83065afbad3bb02c3c464b71f1b3f
# good: [89f081730c49a1d3b46359aa0054e6b3b80f47e4] libceph: use sizeof_footer() more
git bisect good 89f081730c49a1d3b46359aa0054e6b3b80f47e4

[Michael Ellerman (michael-ellerman)]

Thanks.

I see very little left between good and bad which could explain the bug going away. So I'm wondering if the bisect has gone off the rails.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
5ee61e95b6b33c82f6fa1382585faed66aa01245

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Diane Brent (drbrent)]

Can we get answer to #38. If we have the test that caused the failure we will try to reproduce here.

[Mike Rushton (leftyfb)]

The memory_stress_ng test is part of the plainbox-provider-checkbox package which can be installed from the hardware-certification PPA:

https://code.launchpad.net/~hardware-certification/+archive/ubuntu/public

The full path once installed is:
/usr/lib/plainbox-provider-checkbox/bin/memory_stress_ng

[Mike Rushton (leftyfb)]

Commit failed:
ee61e95b6b33c82f6fa1382585faed66aa01245

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
ddc8f6feec76b5deea8090db015920a283006044

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

Commit failed:
ddc8f6feec76b5deea8090db015920a283006044

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
4f1b50c3e3082b31c94cee2b897bd9f5d0f3e7c8

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Mike Rushton (leftyfb)]

Commit failed:
4f1b50c3e3082b31c94cee2b897bd9f5d0f3e7c8

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
698f415cf5756e320623bdb015a600945743377c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1573062

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Balbir Singh (bsingharora)]

I have 14.04 installed with 4.4.0-28 and I can see the following

In the bad case

1. OOM'ing of stress-ng-brk is slow, I can see it making progress -- see tasks being scheduled/console output and sysrq output on Ctrl-o h
2. stress-ng-brk is trying to make progress in OOM, but is heavily contenting for what seems like lru lock
3. The network driver is trying to serve softirq's and fails to process them as allocation fails and does a dump_stack()

In the other case (based on logs and limited testing - kernel 698f415cf5756e320623bdb015a600945743377c for me)
1. OOM proceeds quickly and stress-ng-brk is OOM'd frequently
2. At some point when all stress-ng-brk seem to be OOM'd the test completes

I think the test relies on all stress-ng-brk's to OOM before considering completion (I could be wrong), but in this case progress is slow, but the system is responding. I am going to try the kernel listed here

We believe that the test may require multiple iterations of running for check for consistency of commits.

At this point I think the test will progress and needs longer and progress is slow. I think the expectation is that it needs to complete faster; the system is not lock'd up completely as far as I can tell.

[Balbir Singh (bsingharora)]

Sorry the 14.04 should be 16.04 in comment #61

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-01 02:36 EDT-------
What is the criteria for forward progress of the stress?

I did a quick check for what processes are OOM'd

In new kernel

2 apport
1 cron
1 dhclient
1 gmain
2 in:imklog
1 (journald)
1 kworker/u160:4
1 rs:main
1 stress-ng
972 stress-ng-bighe
157 stress-ng-brk
10 swapper/1
2 swapper/16
17 swapper/2
39 swapper/40
15 swapper/41
1 swapper/65
3 systemd
3 systemd-cgroups
10 systemd-journal
1 systemd-logind

In the 14.04 kernel

1 dhclient
1 in:imklog
1 in:imuxsock
3 irqbalance
1 jbd2/sda2-8
1 kworker/u160:1
1 stress-ng
226 stress-ng-brk
32 swapper/16
3 swapper/23
2 swapper/31
3 swapper/32
22 swapper/46
5 swapper/55
33 swapper/56
3 swapper/63
6 swapper/64
18 swapper/7
6 swapper/72
7 swapper/8

We changed the OOM killer in 4.6 (see https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=aac453635549699c13a84ea1456d5b0e574ef855). Looks like we have good behaviour with 4.6 which could be a result of the change. I am yet to look at the source of memstress_ng, but if the
processes selected for OOM impact the result of the test, we could have a probable explanation. It will also be interesting to continue the bisect and see where we end up.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-06 20:56 EDT-------
From what I can see the following is the root cause of the issue

cgroup_threadgroup_rwsem almost serializes accesses on the system

1. stress-ng-brk has cgroup_threadgroup_rwsem held in read mode via copy_process() and does a schedule_timeout() from __alloc_pages_nodemask() which never seems to return from schedule_timeout()

00003fff96ceeab0 0 2799 2701 0x00040002
[ 4401.831972] Call Trace:
[ 4401.831973] [c000000ee71433c0] [c000000ee7143400] 0xc000000ee7143400 (unreliable)
[ 4401.831975] [c000000ee7143590] [c000000000017c64] __switch_to+0x204/0x360
[ 4401.831977] [c000000ee71435e0] [c000000000bb917c] __schedule+0x40c/0xe70
[ 4401.831979] [c000000ee71436a0] [c000000000bb9c34] schedule+0x54/0xd0
[ 4401.831981] [c000000ee71436d0] [c000000000bc0524] schedule_timeout+0x384/0x4f0
[ 4401.831983] [c000000ee7143800] [c00000000027de1c] __alloc_pages_nodemask+0xd0c/0xf40
[ 4401.831985] [c000000ee7143a10] [c0000000002e8d40] alloc_pages_current+0xc0/0x240
[ 4401.831988] [c000000ee7143a70] [c000000000056b6c] page_table_alloc+0xcc/0x1e0
[ 4401.831989] [c000000ee7143ac0] [c0000000002b5824] __pte_alloc+0x54/0x1e0
[ 4401.831991] [c000000ee7143b10] [c0000000002b8584] copy_page_range+0x754/0x8f0
[ 4401.831993] [c000000ee7143c40] [c0000000000bcee4] copy_process.isra.6+0x1834/0x1ab0
[ 4401.831995] [c000000ee7143d60] [c0000000000bd33c] _do_fork+0xac/0x980
[ 4401.831997] [c000000ee7143e30] [c00000000000946c] ppc_clone+0x8/0xc

[ 4401.861569] cfs_rq[23]:/user.slice
[ 4401.861570] .exec_clock : 1725230.642232
[ 4401.861571] .MIN_vruntime : 0.000001
[ 4401.861572] .min_vruntime : 1154678.434341
[ 4401.861573] .max_vruntime : 0.000001
[ 4401.861573] .spread : 0.000000
[ 4401.861574] .spread0 : -97866589.605918
[ 4401.861575] .nr_spread_over : 11
[ 4401.861575] .nr_running : 0

[ 4401.862187] stress-ng-brk 2799 1154678.007061 854611 120 688670.967816 1148995.803148 2318289.407734 0 0 /user.slice

2. Since cgroup_threadgroup_rwsem is grabbed, we are unable to make any processes exit

[ 4177.396262] Showing all locks held in the system:
[ 4177.396263] 4 locks held by systemd/1:
[ 4177.396268] #0: (sb_writers#9){.+.+.+}, at: [<c000000000328810>] __sb_start_write+0x100/0x130
[ 4177.396272] #1: (&of->mutex){+.+.+.}, at: [<c0000000003e76ec>] kernfs_fop_write+0x7c/0x1f0
[ 4177.396275] #2: (cgroup_mutex){+.+.+.}, at: [<c0000000001b0b8c>] cgroup_kn_lock_live+0x14c/0x280
[ 4177.396278] #3: (&cgroup_threadgroup_rwsem){++++++}, at: [<c00000000013aa10>] percpu_down_write+0x50/0x180

I think at #3, we are waiting for all readers to exit cgroup_threadgroup_rwsem, this further blocks exiting threads

[ 4177.396548] #0: (&cgroup_threadgroup_rwsem){++++++}, at: [<c0000000000d9b00>] exit_signals+0x50/0x1a0
[ 4177.396548] 1 lock held by kworker/dying/1348:
[ 4177.396551] #0: (&cgroup_threadgroup_rwsem){++++++}, at: [<c0000000000d9b00>] exit_signals+0x50/0x1a0
[ 4177.396552] 1 lock held by kworker/dying/1919:
[ 4177.396555...

[Joseph Salisbury (jsalisbury)]

I built a Xenial test kernel[0] with the fix[1] suggested in comment #64. The fix was mentioned upstream[2].

That particular patch has not landed in mainline yet, but it was also cc'd to stable for all 4.4+ kernels.

This commit can be added as a SAUCE patch if we find that it resolves this bug, it will be SAUCE until the real patch is applied through the normal stable update process.

Can you test this kernel and see if it resolves this bug? Note, with this test kernel you need to install both the linux-image and linux-image-extra .deb packages.

[0] http://kernel.ubuntu.com/~jsalisbury/lp1573062/
[1] https://patchwork.kernel.org/patch/8902401/
[2] https://lkml.org/lkml/2016/4/17/56

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-10 23:00 EDT-------
No luck with the new build shared (just 1 run, I'll try more runs).. More debugging in progress as well

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-11 00:59 EDT-------
Can you please provide links to the sources as well, just to do a quick diff against the 4.5 working git?

Have we made further progress on bisect?

[bugproxy (bugproxy)]

Default Comment by Bridge

[Mike Rushton (leftyfb)]

kern.log attached to show additional failures with patched kernel

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-12 04:14 EDT-------
I backported the oom-reaper changes from v4.5 and I've had good runs so far (2 runs with machine returning to console)

I took aac453635549699c13a84ea1456d5b0e574ef855 + next 7 patches and removed unsupported bits. I also took the changes for schedule_timeout_idle() + memcontrol changes I pointed out earlier.

[Leann Ogasawara (leannogasawara)]

Hi,

Joe is away this week and we'd like to keep this bug moving forward. Based on comment #70, we'd like to build a new test kernel to try internally. However, we want to ensure we've applied the exact same changes/patches as noted in comment #70. Could we be pointed at a git repo with the patches? Or could they be attached here to the bug? Thanks in advance.

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2016-07-13 07:53 EDT-------

These are fixes from 4.5 mostly OOM reaper + changes for compilation during the backport

It also includes schedule_timeout_idle() and the memcontrol.c fixes

[Kamal Mostafa (kamalmostafa)]

Xenial (4.4.0-32.51~lp1573062) ppc64el test kernel comprised of 4.4.0-31.50 plus the changes from balbirs' comment #72's oom_reaper_4.4.diff*:

    http://people.canonical.com/~kamal/lp1573062/

*The source branch for that test kernel, with the original mainline commits (and LKML patch) split back out (resulting in a delta identical to comment #72 oom_reaper_4.4.diff):

    https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-13 23:04 EDT-------
I also added af8e15cc85a253155fdcea707588bf6ddfc0be2e to my diff, just FYI

[Kamal Mostafa (kamalmostafa)]

Per comment #74, I've now also added af8e15cc85a253155fdcea707588bf6ddfc0be2e ("oom, oom_reaper: do not enqueue task if it is on the oom_reaper_list head") to the test kernel and source branch.

Xenial (4.4.0-32.51~lp1573062.1) ppc64el test kernel:

  http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

  https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062

[Mike Rushton (leftyfb)]

4.4.0-32-generic-51~lp1573062.1 failed.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

[Joseph Salisbury (jsalisbury)]

It looks like there are conflicting test results with the 4.4.0-32-generic-51~lp1573062.1 test kernel.

@balbirs, can you confirm the test was run enough times to reproduce the bug?

@Mike Rushton, can you confirm that you can make the test fail multiple times with that kernel?

[Mike Rushton (leftyfb)]

I'm running the tests as we speak across 2 LPAR's and 2 openpower PowerNV installations. One of the open power boxes has 256G of memory so it takes up to 7 hours for a test to pass or to be determined failed.

[Mike Rushton (leftyfb)]

Attached is the kern.log from the Firestone PowerNV server. This one is failing with stack traces which only started after we started testing patched kernels.

The Habanero PowerNV failed with the typical locked up console and OOM's.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

[Joseph Salisbury (jsalisbury)]

@balbirs, would it be possible for you to send the individual .patch files, so I can apply them with "git am"? That will confirm we are using the same code base.

You can generate them with something like:

'git format-patch -k HEAD~10'

Change the number '10' to the number of commits you added with your changes/diffs. Alternatively, if you have your repo somewhere I can clone from, I can do that.

I can then build a test kernel with your patches, and build a test kernel with lockdep enabled.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

[Joseph Salisbury (jsalisbury)]

@balbirs, are you saying the repo you cloned from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 differs from your diffs?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

[Joseph Salisbury (jsalisbury)]

Thanks for the update. I'll clone the source as well and rebuild the binaries for testing.

[Joseph Salisbury (jsalisbury)]

I cloned kamal's repository and built a test kernel. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1573062/

balbirs and Mike, can you test this kernel and see if it still exhibits the bug?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

------- Comment From <email address hidden> 2016-07-25 09:09 EDT-------
In the meanwhile, any updates on the bisect? I was hoping we could do both things (RCA and bisect) in parallel
...

[Joseph Salisbury (jsalisbury)]

The bisect was stopped because the first good kernel was found to actually be bad. We can restart the bisect if we can find a good and bad kernel again.

@Mike, could you test the v4.7 kernel and see if the bug is fixed in it? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7/

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

------- Comment From <email address hidden> 2016-07-25 09:09 EDT-------
In the meanwhile, any updates on the bisect? I was hoping we could do both things (RCA and bisect) in parallel
...

[Mike Rushton (leftyfb)]

mainline kernel 4.7 failed.

[Mike Rushton (leftyfb)]

Output of 4.7 mainline kernel testing failure

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-17 21:43 EDT-------
I tried the kernel at http://people.canonical.com/~kamal/lp1573062/lp1573062.1/ and it worked fine for me

------- Comment From <email address hidden> 2016-07-19 01:04 EDT-------
Looks like I got a failure with the run on http://people.canonical.com/~kamal/lp1573062/lp1573062.1/

But with my diff + 4.4.0 source from apt-source, I can always get the the following command to succeed.

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

I've tried three times with my diff (all success) and twice with the kernel @ ~kamal (one failure and one success). I've not tried the longer 7 hour run

------- Comment From <email address hidden> 2016-07-19 01:37 EDT-------
In the kern.log posted, it looks like the problem has moved to

rwsem_wake+0xcc/0x110
up_write+0x78/0x90
unlink_anon_vmas+0x15c/0x2c0

A bunch of threads are stuck on rwsem_wake -- spinning on the sem->wait_lock. I can see a whole bunch of exiting stress-ng-mmapf stuck on this lock, spinning. I'll double check this. Can we get a build with lockdep enabled? I am unable to reproduce this issue at my end with the diff applied on my machine at the moment

------- Comment From <email address hidden> 2016-07-19 19:51 EDT-------
I am cloning the sources to debug further

------- Comment From <email address hidden> 2016-07-19 23:52 EDT-------
I cloned the kernel from https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux/+git/xenial/log/?h=lp1573062 and built with the machine config specified from /boot/config. I also verified the diff matches my changes.

I ran

timeout -s 9 $end_time stress-ng --aggressive --verify --timeout $runtime --brk 0

twice

Both the times, the test did the right thing. Could someone verify if

(a) The smaller subset works fine?
(b) The larger test fails, if so, can we get a run with lockdep

I was just testing for the command line above and I could see a difference with those patches.

------- Comment From <email address hidden> 2016-07-21 20:14 EDT-------
No, the diff matches, sorry for the confusion, but here is what I said

"I also verified the diff matches my changes"

In summary, here is what I did

1. cloned the sources
2. built locally on my machine
3. Ran stress-ng with recommended parameters
4. The test succeeded, got back the console

Did four runs and I got back the console each time

However with the provided binaries

Step 3 (stress-ng) failed for me once in two runs

------- Comment From <email address hidden> 2016-07-25 08:08 EDT-------
Strange, I am able to reproduce the issue with the provided binaries, but not when I build it. I am not doing a deb build, but just a make -j64 with the config from /boot for 4.4.0-28. The problem could be at my end, but I am a little concerned.

I also noticed that if I am interacting with the system during runs, it succeeds, frequently checking if the console is active (enters and control-o-h). I am going to see if I can get a repro again and debug further.

------- Comment From <email address hidden> 2016-07-25 09:09 EDT-------
In the meanwhile, any updates on the bisect? I was hoping we could do both things (RCA and bisect) in parallel
...

[Joseph Salisbury (jsalisbury)]

I built a Xenial test kernel with the latest patch posted to mm by balbirs. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1573062/

@Mike, can you test this kernel when you have a chance?

[Michael Hohnbaum (hohnbaum)]

Balbir,

Both Mike and Joe are out the rest of this week on holiday. Any chance you could give this kernel a spin?

                 Michael

[Balbir Singh (bsingharora)]

I just looked at the directory and I can find just the arm64 kernel. Could you please confirm if I am looking at the right thing and at the right place?

[Joseph Salisbury (jsalisbury)]

Sorry I built arm64. I'll build ppc64el now.

[Joseph Salisbury (jsalisbury)]

The test kernel is now available in the same location:
http://kernel.ubuntu.com/~jsalisbury/lp1573062/

[Joseph Salisbury (jsalisbury)]

Also note, with this test kernel you have to install both the linux-image and linux-image-extra .deb packages.

[Balbir Singh (bsingharora)]

At my end, I ran two runs with success. More runs in progress

[Balbir Singh (bsingharora)]

Had several other runs of success. I would like to see runs from others as well.

[Balbir Singh (bsingharora)]

Can I quickly check if the oom_reaper patches are there in the built kernel or is it just the fix I posted?

[Joseph Salisbury (jsalisbury)]

The test kernel in comment #103 did not have the oom_reaper patches. It only had the fix you posted.

[Kalpana S Shetty (kalshett)]

I have had 5 successful runs so far with below kernel. however I keep see OOM call traces.

root@ltc-haba1:~/log# uname -a
Linux ltc-haba1 4.4.0-34-generic #53~lp1573062PATCHED SMP Wed Aug 10 23:59:53 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

[Mike Rushton (leftyfb)]

I had this fail on 2 openpower boxes. I have attached the output from the test.

[Kalpana S Shetty (kalshett)]

I have been trying to recreate this issue on my Haba-LC system and not able to do so.
The difference I see in total memory installed on my system against your system i.e 32Gig.

Total memory is 256 GiB -----------------------------> my Haba-LC.
Constant run time is 300 seconds per stressor
Variable run time is 2860 seconds per stressor
Estimated total run time is 396 minutes

Is there any way I can run the test for less memory config, the system I have got is 256Gig mem.

[Mike Rushton (leftyfb)]

Kalpana:

The above failed test was run on only 1 of the possible 4 machines we have been testing with for months now. The other openpower server is a Habanero with 256G. I am running tests on that as we speak and going to include the kern.log as well.

Just to clarify, the memory test I'm running is /usr/lib/plainbox-provider-checkbox/bin/memory_stress_ng

[Balbir Singh (bsingharora)]

I am unable to reproduce the failure either, but your system with 32G and 128 threads seems like the test would start 128 hogs each hogging up 32GB. How much swap do you have on them? Could you post the dmesg to see what failed and the logs around it? It looks like the stack stressor failed.

[Mike Rushton (leftyfb)]

This is a failed test from today on the Habanero. Find the test output and kern.log attached:

ubuntu@binacle:~/4.4.0-34-generic-53~lp1573062PATCHED$ free -m
              total used free shared buff/cache available
Mem: 261533 621 259912 20 1000 259938
Swap: 8191 0 8191

[Mike Rushton (leftyfb)]

kern.log from the Habanero 4.4.0-34-generic-53~lp1573062PATCHED test

[Mike Rushton (leftyfb)]

This is a failed test from today on the Firestone. Find the test output and kern.log attached:

ubuntu@gulpin:~/4.4.0-34-generic-53~lp1573062PATCHED$ free -m
              total used free shared buff/cache available
Mem: 32565 342 30092 21 2131 30918
Swap: 8191 0 8191

[Mike Rushton (leftyfb)]

kern.log from the Firestone 4.4.0-34-generic-53~lp1573062PATCHED test

[Balbir Singh (bsingharora)]

These logs are something I've not seen here in my testing. This shows that we are stuck doing an up_write() on root->rwsem in the anon_vma path. It looks like we are contending on the rwsem's sem->wait_lock. I don't have a reproduction of this issue, it will be interesting to examine what is causing the heavy contention

[Mike Rushton (leftyfb)]

Out of the 4 tests last night (Habanero(NV), Firstone(NV), Tuleta(NV), Alpine(VM) only the Firestone failed. Attached is the output from the test, kern.log and dmesg output I was running in a while loop during testing.

[Mike Rushton (leftyfb)]

kern.log from failed test on Firestone

[Mike Rushton (leftyfb)]

dmesg from failed test on Firestone

[Jeff Lane  (bladernr)]

Just an additional data point, I ran this test (using Mike's script in a loop) on other architectures:

amd64 bare metal: 30 times 4.4.0, no failures
amd64 virtual machine: 30 times, 4.4.0, no failures

s390 4GB RAM: 40 times on both 4.4.0-31 and 4.7.0, no failures
s390 20GB RAM: 40 times on both 4.4.0-31 and 4.7.0, no failures.

I'm still running on arm64 and waiting on final results.

[Balbir Singh (bsingharora)]

Thanks Jeff. I see that the ARM64 might have failed - https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1610320

Can we know the git commit id that fixed the ARM64 failure in mainline?

BTW, could you please share the full machine configurations - threads+RAM+swap for each of the other architectures you ran this on?

[bugproxy (bugproxy)]

This comment is to inform you that mirroring has been disabled between Launchpad bug 1573062 and LTC bug 141723 due to the size of this LP bug attachments.

If you want to make this LP bug bridgeable again, I suggest you to remove unwanted attachments or perhaps to compress some of the existing attachments (several large uncompressed text log files currently exist).

[Balbir Singh (bsingharora)]

I just posted another patch @ http://<email address hidden>/msg1219903.html, I am testing this patch at the moment.

[Kalpana S Shetty (kalshett)]

Joseph: Is it possible for you to post test kernel with Balbir's patches ?
and place them at earlier location - http://kernel.ubuntu.com/~jsalisbury/lp1573062/

[Mike Rushton (leftyfb)]

Attached is the cultivation of all the kernel testing we have done.

[Balbir Singh (bsingharora)]

Thank you for the excellent summary. Questions

1. Can we get the configurations of the machines.
2. The first column is the number of times the test ran?
3. I see that 4.4.0-31-generic-50-Ubuntu passed on all machines across several runs, is that true?
4. Did any of the tests result in system hang? Can we find out from the summary?

Would it be fair to assume that tests when run against 4.4.0-31-generic-50-Ubuntu would pass again and we should work off of that?

My interest is in 4.4.0-34-generic-53~lp1573062PATCHED, With that I notice that gulpin saw failures in mmapfork and probably a hang there -- for which I posted a scheduler try_to_wake_up fix upstream. Generally binacle faces brk stress test failures -- it will be interesting to see its machine configuration and why the test failed

One observation at my end is that we should reboot between runs as I think some tests can kill important tasks in the system and I am not sure if there is a guarantee that the system is able to carry on correct operation recovering from all the tasks being re-spawned after OOM for example.

[Mike Rushton (leftyfb)]

Sorry, I should have taken out the 4.4.0-31 tests. That test was not using stress-ng but our original memory stress test.

[Balbir Singh (bsingharora)]

Can we please get answers to 1, 2 and 4 for comment #128. Also Kalpana has a request for a new kernel build.

Thanks,

[Mike Rushton (leftyfb)]

1. Can we get the configurations of the machines.

XML files for all 4 machines with hardware information from the certification process will be attached in the next few comments

2. The first column is the number of times the test ran?

Yes

4. Did any of the tests result in system hang? Can we find out from the summary?

All failed tests failed with a complete system hang. The only way to recover would be to reboot.

[Joseph Salisbury (jsalisbury)]

Balbir, Kalpana,

I can build the kernel for you. Can you post a list of all the patches you would like included? That way we ensure they are all included.

[Balbir Singh (bsingharora)]

Can we build the latest 4.8, may be we should wait for 4.8-rc7. I've got all the fixes upstream, with the latest being 135e8c9250dd5c8c9aae5984fde6f230d0cbfeaf

[Joseph Salisbury (jsalisbury)]

Sure, we can wait for 4.8-rc7. It should be available on Monday. I'll post a link to the download location once it's available.

[Balbir Singh (bsingharora)]

FYI, I think the patch made it to 4.4 stable as well

[Joseph Salisbury (jsalisbury)]

Balibir, the v4.8 final kernel is now available and can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/

The latest upstream stable 4.4 kernel is also avaiable from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.23

[Mike Rushton (leftyfb)]

output of stress_ng test running on Power 8 Tuleta LPAR

[Mike Rushton (leftyfb)]

dmesg output while stress_ng test was running on Power 8 Tuleta LPAR

[Mike Rushton (leftyfb)]

`ps -ax` output while stress_ng test was running on Power 8 Tuleta LPAR

[Colin Ian King (colin-king)]

We're getting zombies here which aren't being reaped:

130428 ? Z 0:00 [stress-ng-brk] <defunct>
130432 ? Z 0:00 [stress-ng-brk] <defunct>
130434 ? Z 0:00 [stress-ng-brk] <defunct>
130436 ? Z 0:00 [stress-ng-brk] <defunct>

The reason for this is that memory stressors like brk have a parent that forks off a child. The child performs the stressing and if it gets OOM'd the parent can spawn off another stressor. So I think the SIGKILL on the stress-ng brk stressor is killing the parent bug the child (which is still holding onto a load of memory on the heap) is not being waited for and hence is in a memory hogging zombie state. We may be in a pathologically memory hogging state because the zombies may be holding brk regions that are swapped out to disk due to memory pressure and we're hitting a low-memory state which is not being cleared up.

I suggest modifying the test bash script as follows:

1. run stress-ng with -k flag (so that all the processes have the same stress-ng name)
2. kill with ALRM first
3. then kill with KILL all the stress-ng processes after a small grace period.
4. report on unkillable stressors

refer to the changes I made to https://launchpadlibrarian.net/296974522/disk_stress_ng

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-12-20 22:55 EDT-------
Could the Ubuntu team check if this is still an issue with the 4.8 kernel?

[Jeff Lane  (bladernr)]

Ok, so we're now using stress-ng 0.07.21 which resolved a lot of issues and have also adopted Colin's suggestions into our wrapper script to modify how stress-ng is being called and culled.

This seems to resolve the problems that led to this.

[Mike Rushton (leftyfb)]

This issue, or similar symptoms doesn't seem to be fixed. This is with stock kernel, stock stress-ng but with the suggested changes to the memory_stress_ng test script. The machine locks up completely maybe 1 out of 5-10 runs. See attached dmesg for errors.

Kernel: 4.4.0-64-generic-85-Ubuntu
stress-ng Version: 0.07.21-1~ppa

[Jeff Lane  (bladernr)]

OK, I have a Firestone available, so I'm re-trying this with a cycle of 10 runs to see what happens. Currently using 16.04 w/ HWE-Edge (4.10)

[Jeff Lane  (bladernr)]

OK... so I tried three runs of the stress-ng memory test (that has been used successfully on every other system we certify for 16.04) and so far, two of three runs have resulted in the system locking up and a bunch of stack trace data dumped to console.

In both cases, I had to power cycle the system to reboot it into a usable state.

All three attempts were 16.04 w/ hwe-edge (4.10) deployed by MAAS.

System info:

Linux oil-entei 4.10.0-20-generic #22~16.04.1-Ubuntu SMP Thu Apr 20 10:30:58 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

stress-ng:
  Installed: 0.07.21-1~ppa
(We keep an updated version of stress-ng in the cert PPA, it's not modified from Colin's code)

I've attached logs (kernel and syslog, other info) in a tarball to this.

[Jeff Lane  (bladernr)]

resetting all the tasks to confirmed, rather than Incomplete as I've tested and am still waiting for a resolution or whatever.

[Andy Whitcroft (apw)]

This bug was nominated against a series that is no longer supported, ie yakkety. The bug task representing the yakkety nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.