vfork in ubuntu_stress_smoke test failed on KVM node with B-KVM kernel

Bug #1788385 reported by Po-Hsu Lin on 2018-08-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Stress-ng
High
Colin Ian King
ubuntu-kernel-tests
Undecided
Unassigned
linux-kvm (Ubuntu)
Undecided
Unassigned
Nominated for Bionic by Po-Hsu Lin

Bug Description

Fail rate 2 out of 2, failed on a kvm node "gummo", with 1G ram

uname -vr : 4.15.0-1020-kvm #20-Ubuntu SMP Thu Aug 16 16:46:26 UTC 2018
Bionic 18.04
arch: amd64

DEBUG - 11:24:24 DEBUG| [stdout] vfork STARTING
DEBUG - 11:24:24 DEBUG| [stdout] vfork STARTING
DEBUG - 11:24:26 DEBUG| [stdout] vfork RETURNED 2
DEBUG - 11:24:26 DEBUG| [stdout] vfork RETURNED 2
DEBUG - 11:24:26 DEBUG| [stdout] vfork FAILED
DEBUG - 11:24:26 DEBUG| [stdout] vfork FAILED
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] 2 processors online, 2 processors configured
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] 2 processors online, 2 processors configured
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: info: [1437] dispatching hogs: 4 vfork
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: info: [1437] dispatching hogs: 4 vfork
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] cache allocate: reducing cache level from L3 (too high) to L2
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] cache allocate: reducing cache level from L3 (too high) to L2
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] cache allocate: default cache size: 4096K
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] cache allocate: default cache size: 4096K
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] starting stressors
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] starting stressors
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1438] stress-ng-vfork: started [1438] (instance 0)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1438] stress-ng-vfork: started [1438] (instance 0)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] 4 stressors spawned
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] 4 stressors spawned
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1439] stress-ng-vfork: started [1439] (instance 1)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1439] stress-ng-vfork: started [1439] (instance 1)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1440] stress-ng-vfork: started [1440] (instance 2)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1440] stress-ng-vfork: started [1440] (instance 2)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1441] stress-ng-vfork: started [1441] (instance 3)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1441] stress-ng-vfork: started [1441] (instance 3)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1441] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1441] stress-ng-vfork: exited [1441] (instance 3)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1441] stress-ng-vfork: exited [1441] (instance 3)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1440] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1440] stress-ng-vfork: exited [1440] (instance 2)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1440] stress-ng-vfork: exited [1440] (instance 2)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1438] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1438] stress-ng-vfork: exited [1438] (instance 0)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1438] stress-ng-vfork: exited [1438] (instance 0)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: fail: [1439] stress-ng-vfork: fork failed
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] info: 5 failures reached, aborting stress process
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1438 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1438 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1438] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1438] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1439] stress-ng-vfork: exited [1439] (instance 1)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1439] stress-ng-vfork: exited [1439] (instance 1)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1439 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1439 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1439] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1439] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1440 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1440 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1440] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1440] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1441 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: error: [1437] process 1441 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1441] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: debug: [1437] process [1441] terminated
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: info: [1437] unsuccessful run completed in 1.49s
DEBUG - 11:24:26 DEBUG| [stdout] stress-ng: info: [1437] unsuccessful run completed in 1.49s

Colin Ian King (colin-king) wrote :

Do you mind providing information on the architecture and kernel version + release that is being tested.

Po-Hsu Lin (cypressyew) wrote :

Sorry was in a hurry, info updated in the bug description.

description: updated
description: updated
Po-Hsu Lin (cypressyew) wrote :

Tested multiple time on the same node with both Bionic generic / kvm kernel:

4.15.0-32.35: failed 0/3
4.15.0.1019.19: failed 2/3

Test was executed 3 times consecutively without reboot, output from syslog when it fails:
Aug 23 06:18:59 gummo kernel: Adding 1478652k swap on /swap.img. Priority:-2 extents:3 across:1830908k
Aug 23 06:18:59 gummo kernel: Adding 1048572k swap on /home/ubuntu/autotest/client/tmp/ubuntu_stress_smoke_test/src/stress-ng/swap.img. Priority:-3 extents:3 across:7340032k
Aug 23 06:18:59 gummo stress-ng: invoked with './stress-n' by user 0
Aug 23 06:18:59 gummo stress-ng: system: 'gummo' Linux 4.15.0-1019-kvm #19-Ubuntu SMP Fri Aug 10 10:24:21 UTC 2018 x86_64
Aug 23 06:18:59 gummo stress-ng: memory (MB): total 996.66, free 7.01, shared 0.50, buffer 11.73, swap 2467.99, free swap 2467.99
Aug 23 06:18:59 gummo stress-ng: info: [823] dispatching hogs: 4 vfork
Aug 23 06:19:01 gummo stress-ng: fail: [827] stress-ng-vfork: fork failed
Aug 23 06:19:01 gummo stress-ng: fail: [825] stress-ng-vfork: fork failed
Aug 23 06:19:01 gummo stress-ng: message repeated 4 times: [ fail: [825] stress-ng-vfork: fork failed]
Aug 23 06:19:01 gummo stress-ng: fail: [827] stress-ng-vfork: fork failed
Aug 23 06:19:01 gummo stress-ng: message repeated 3 times: [ fail: [827] stress-ng-vfork: fork failed]
Aug 23 06:19:02 gummo stress-ng: fail: [824] stress-ng-vfork: fork failed
Aug 23 06:19:02 gummo stress-ng: message repeated 4 times: [ fail: [824] stress-ng-vfork: fork failed]
Aug 23 06:19:02 gummo stress-ng: fail: [826] stress-ng-vfork: fork failed
Aug 23 06:19:02 gummo stress-ng: error: [823] process 824 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
Aug 23 06:19:02 gummo stress-ng: error: [823] process 825 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
Aug 23 06:19:04 gummo stress-ng: fail: [826] stress-ng-vfork: fork failed
Aug 23 06:19:04 gummo stress-ng: message repeated 3 times: [ fail: [826] stress-ng-vfork: fork failed]
Aug 23 06:19:04 gummo stress-ng: error: [823] process 826 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
Aug 23 06:19:04 gummo stress-ng: error: [823] process 827 (stress-ng-vfork) terminated with an error, exit status=1 (stress-ng core failure)
Aug 23 06:19:04 gummo stress-ng: info: [823] unsuccessful run completed in 4.51s

Colin Ian King (colin-king) wrote :

This is not a kernel error; it's a bug in stress-ng when running with the --verify option on systems where memory is low and EAGAIN or ENOMEM occurs. I've fixed this in stress-ng with commit:

http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=8ab0ac7ce995517c0cb10dddddb60bee5a19b7b8

Do you mind running a re-test to see if this fixes the issue. Apologies for this stress-ng bug. My bad.

Changed in stress-ng:
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
status: New → In Progress
Po-Hsu Lin (cypressyew) wrote :

Passed with 3 run on the same node with the updated repo.
Thank you!

Changed in ubuntu-kernel-tests:
status: New → Fix Released
Changed in linux-kvm (Ubuntu):
status: New → Invalid
Po-Hsu Lin (cypressyew) on 2018-08-23
Changed in stress-ng:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers