pthread in stress-smoke test failed on GCP X-4.15

[Po-Hsu Lin]

 4.15.0-1027-gcp #28~16.04.1-Ubuntu
 Only spotted on node n1-standard-8

 pthread FAILED
 stress-ng: debug: [19921] 1 processor online, 1 processor configured
 stress-ng: info: [19921] dispatching hogs: 4 pthread
 stress-ng: debug: [19921] cache allocate: default cache size: 56320K
 stress-ng: debug: [19921] starting stressors
 stress-ng: debug: [19921] 4 stressors spawned
 stress-ng: debug: [19925] stress-ng-pthread: started [19925] (instance 3)
 stress-ng: debug: [19924] stress-ng-pthread: started [19924] (instance 2)
 stress-ng: debug: [19923] stress-ng-pthread: started [19923] (instance 1)
 stress-ng: debug: [19922] stress-ng-pthread: started [19922] (instance 0)
 stress-ng: debug: [19923] stress-ng-pthread: exited [19923] (instance 1)
 stress-ng: debug: [19922] stress-ng-pthread: exited [19922] (instance 0)
 stress-ng: debug: [19921] process [19922] terminated
 stress-ng: debug: [19924] stress-ng-pthread: exited [19924] (instance 2)
 stress-ng: fail: [19924] stress-ng-pthread: mutex unlock failed, errno=22 (Invalid argument)
 stress-ng: debug: [19921] process [19923] (stress-ng-pthread) terminated on signal: 11 (Segmentation fault)
 stress-ng: debug: [19921] process [19923] terminated
 stress-ng: debug: [19921] process [19924] terminated
 stress-ng: debug: [19921] process [19925] (stress-ng-pthread) terminated on signal: 11 (Segmentation fault)
 stress-ng: debug: [19921] process [19925] terminated
 stress-ng: info: [19921] unsuccessful run completed in 5.05s

[Po-Hsu Lin (cypressyew)]

Reproduce rate 3 out of 3 attempts.

[Colin Ian King (colin-king)]

Do we have any more logs of the run, e.g. kernel messages etc?

[Po-Hsu Lin (cypressyew)]

Sorry I can't get that without access to the node.

The test itself won't print any kernel messages.

[Colin Ian King (colin-king)]

Added some extra debug and some more sanity checking. Looks like on a single CPU the pthread reaping was not unlocking a mutex causing the issue.

Fixes committed:
https://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=e5ecba459754d4bdb3e858caaaf41a87e485c8ce
"stress-pthread: make error messages a bit more informative"

and
https://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=a50c7466634e194e3169bc63591a86342cb10c67
stress-pthread: add better error recovery and locking prevention checks