stress-ng failure causing checkbox-ng to traceback

[Jeff Lane ]

I'm trying to run canonical-certification-server on zKVM, zVM and an LPAR on Z.

On zKVM, stress-ng works fine. However on zVM and the LPAR, stress-ng is failing it's testing. I filed this bug to address this directly in stress-ng:

https://bugs.launchpad.net/ubuntu/+source/stress-ng/+bug/1569468

So, when this is run in the context of checkbox-ng, it's causing checkbox-ng to throw a traceback and exit violently:

-------------[ Running job 50 / 61. Estimated time left: unknown ]--------------
------------------------[ Stress test of system memory ]------------------------
ID: 2013.com.canonical.certification::memory/memory_stress_ng
Category: 2013.com.canonical.plainbox::memory
... 8< -------------------------------------------------------------------------
Total memory is 4 GiB
Constant run time is 300 seconds per stressor
Variable run time is 340 seconds per stressor
Estimated total run time is 149 minutes

Running stress-ng bsearch stressor for 300 seconds....
stress-ng: info: [54302] dispatching hogs: 4 bsearch
stress-ng: info: [54302] successful run completed in 300.02s (5 mins, 0.02 secs)
return_code is 0
Running stress-ng context stressor for 300 seconds....
stress-ng: info: [54326] dispatching hogs: 4 context
stress-ng: info: [54326] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng hsearch stressor for 300 seconds....
stress-ng: info: [54349] dispatching hogs: 4 hsearch
stress-ng: info: [54349] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng lockbus stressor for 300 seconds....
Try 'stress-ng --help' for more information.
return_code is 1
*****************************************************************
** Error 1 reported on stressor lockbus!)
*****************************************************************
Running stress-ng lsearch stressor for 300 seconds....
stress-ng: info: [54367] dispatching hogs: 4 lsearch
stress-ng: info: [54367] successful run completed in 300.02s (5 mins, 0.02 secs)
return_code is 0
Running stress-ng matrix stressor for 300 seconds....
stress-ng: info: [54387] dispatching hogs: 4 matrix
stress-ng: info: [54387] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng memcpy stressor for 300 seconds....
stress-ng: info: [54404] dispatching hogs: 4 memcpy
stress-ng: info: [54404] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng null stressor for 300 seconds....
stress-ng: info: [54426] dispatching hogs: 4 null
stress-ng: info: [54426] successful run completed in 300.01s (5 mins, 0.01 secs)
return_code is 0
Running stress-ng pipe stressor for 300 seconds....
stress-ng: info: [54455] dispatching hogs: 4 pipe
stress-ng: info: [54455] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng qsort stressor for 300 seconds....
stress-ng: info: [54482] dispatching hogs: 4 qsort
stress-ng: info: [54482] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng stack stressor for 300 seconds....
stress-ng: info: [54497] dispatching hogs: 4 stack
stress-ng: info: [54497] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng str stressor for 300 seconds....
stress-ng: info: [54523] dispatching hogs: 4 str
stress-ng: info: [54523] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng stream stressor for 300 seconds....
stress-ng: info: [54540] dispatching hogs: 4 stream
stress-ng: info: [54540] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng tsearch stressor for 300 seconds....
stress-ng: info: [54560] dispatching hogs: 4 tsearch
stress-ng: info: [54560] successful run completed in 300.04s (5 mins, 0.04 secs)
return_code is 0
Running stress-ng vm-rw stressor for 300 seconds....
stress-ng: info: [54584] dispatching hogs: 4 vm-rw
stress-ng: info: [54584] successful run completed in 300.01s (5 mins, 0.01 secs)
return_code is 0
Running stress-ng wcs stressor for 300 seconds....
stress-ng: info: [54610] dispatching hogs: 4 wcs
stress-ng: info: [54610] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng zero stressor for 300 seconds....
stress-ng: info: [54626] dispatching hogs: 4 zero
stress-ng: info: [54626] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng mlock stressor for 300 seconds....
stress-ng: info: [54647] dispatching hogs: 4 mlock
stress-ng: info: [54647] successful run completed in 300.05s (5 mins, 0.05 secs)
return_code is 0
Running stress-ng mmapfork stressor for 300 seconds....
stress-ng: info: [54668] dispatching hogs: 4 mmapfork
stress-ng: info: [54668] successful run completed in 300.20s (5 mins, 0.20 secs)
return_code is 0
Running stress-ng mmapmany stressor for 300 seconds....
stress-ng: info: [62729] dispatching hogs: 4 mmapmany
stress-ng: info: [62729] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng mremap stressor for 300 seconds....
stress-ng: info: [62763] dispatching hogs: 4 mremap
stress-ng: info: [62763] successful run completed in 300.03s (5 mins, 0.03 secs)
return_code is 0
Running stress-ng shm-sysv stressor for 300 seconds....
stress-ng: info: [62784] dispatching hogs: 4 shm-sysv
stress-ng: info: [62784] successful run completed in 300.01s (5 mins, 0.01 secs)
return_code is 0
Running stress-ng vm-splice stressor for 300 seconds....
stress-ng: info: [62804] dispatching hogs: 4 vm-splice
stress-ng: info: [62804] successful run completed in 300.00s (5 mins, 0.00 secs)
return_code is 0
Running stress-ng malloc stressor for 340 seconds....
stress-ng: info: [62825] dispatching hogs: 4 malloc
stress-ng: info: [62825] successful run completed in 340.10s (5 mins, 40.10 secs)
return_code is 0
Running stress-ng mincore stressor for 340 seconds....
stress-ng: info: [62848] dispatching hogs: 4 mincore
stress-ng: info: [62848] successful run completed in 340.01s (5 mins, 40.01 secs)
return_code is 0
Running stress-ng vm stressor for 340 seconds....
stress-ng: info: [62871] dispatching hogs: 4 vm
stress-ng: info: [62871] successful run completed in 340.02s (5 mins, 40.02 secs)
return_code is 0
Running stress-ng bigheap stressor for 340 seconds....
stress-ng: info: [62900] dispatching hogs: 4 bigheap
stress-ng: info: [62900] successful run completed in 340.39s (5 mins, 40.39 secs)
return_code is 0
Running stress-ng brk stressor for 340 seconds....
stress-ng: info: [62928] dispatching hogs: 4 brk
stress-ng: info: [62928] successful run completed in 340.88s (5 mins, 40.88 secs)
return_code is 0
Running stress-ng mmap stressor for 340 seconds....
stress-ng: info: [62962] dispatching hogs: 4 mmap
stress-ng: info: [62962] unsuccessful run completed in 341.33s (5 mins, 41.33 secs)
return_code is 2
*****************************************************************
** Error 2 reported on stressor mmap!)
*****************************************************************
*******************************************************************
** stress-ng memory test failed; most recent error was 2
*******************************************************************
------------------------------------------------------------------------- >8 ---

                        Application Malfunction Detected

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
    return self._dispatch()
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in _dispatch
    result = ingredient.dispatch(self.context)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 152, in dispatch
    return self._dispatch(context, 0)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 165, in _dispatch
    retval = command.invoked(context)
  File "/usr/bin/checkbox-cli", line 210, in invoked
    self._run_jobs(self.ctx.sa.get_dynamic_todo_list())
  File "/usr/bin/checkbox-cli", line 411, in _run_jobs
    job, self._get_ui_for_job(job))
  File "/usr/bin/checkbox-cli", line 474, in _run_single_job_with_ui_loop
    result_builder = self.ctx.sa.run_job(job.id, ui, False)
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 104, in wrapper
    raise exc
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/assistant.py", line 1222, in run_job
    job, job_state, self._config, ui
  File "/usr/lib/python3/dist-packages/plainbox/impl/runner.py", line 434, in run_job
    return runner(job, job_state, config)
  File "/usr/lib/python3/dist-packages/plainbox/impl/runner.py", line 460, in run_shell_job
    return self._just_run_command(job, job_state, config).get_result()
  File "/usr/lib/python3/dist-packages/plainbox/impl/runner.py", line 804, in _just_run_command
    job, job_state, config, ctrl)
  File "/usr/lib/python3/dist-packages/plainbox/impl/runner.py", line 906, in _run_command
    job, job_state, config, extcmd_popen, ctrl)
  File "/usr/lib/python3/dist-packages/plainbox/impl/runner.py", line 917, in _run_extcmd
    extcmd_popen)
  File "/usr/lib/python3/dist-packages/plainbox/impl/ctrl.py", line 559, in execute_job
    return return_code
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/plainbox/impl/ctrl.py", line 617, in temporary_cwd
    self.on_leftover_files(job, config, cwd_dir, leftovers)
  File "/usr/lib/python3.5/tempfile.py", line 947, in __exit__
    self.cleanup()
  File "/usr/lib/python3.5/tempfile.py", line 951, in cleanup
    _rmtree(self.name)
  File "/usr/lib/python3.5/shutil.py", line 474, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.5/shutil.py", line 408, in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 406, in _rmtree_safe_fd
    dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'tmp-stress-ng-mmap-62964-1'

  Please report a bug including the information from the paragraph above. To
  report the bug visit https://bugs.launchpad.net/checkbox-ng/+filebug

  We are sorry for the inconvenience!

ii checkbox-ng 0.25+bzr4299+pkg38~ubuntu16.04.1 all PlainBox based test runner
ii plainbox-provider-checkbox 0.27+bzr4299+pkg35~ubuntu16.04.1 s390x CheckBox provider for PlainBox
ii python3-checkbox-ng 0.25+bzr4299+pkg38~ubuntu16.04.1 all PlainBox based test runner (Python 3 library)
ii python3-checkbox-support 0.25+bzr4299+pkg27~ubuntu16.04.1 all collection of Python modules used by PlainBox providers
ii plainbox-insecure-policy 0.27+bzr4299+pkg41~ubuntu16.04.1 all policykit policy required to use plainbox (insecure version)
ii plainbox-provider-certification-server 0.26+201604060718~ubuntu16.04.1 all Server Certification provider for Plainbox
ii plainbox-provider-checkbox 0.27+bzr4299+pkg35~ubuntu16.04.1 s390x CheckBox provider for PlainBox
ii plainbox-provider-resource-generic 0.25+bzr4299+pkg27~ubuntu16.04.1 s390x CheckBox generic resource jobs provider
ii python3-plainbox 0.27+bzr4299+pkg41~ubuntu16.04.1 all toolkit for software and hardware testing (python3 module)