Resource Volume has ERROR status. Fault: n/a

[Andrew Kalach]

Description
===========
During 9.0RC2 Certification Testing Sutie run, the "CinderVolumes.create_and_extend_volume " task failed with error:

Resource <Volume: 56f93647-5ffc-44f6-9ae0-e804b39dd475> has ERROR status. Fault: n/a

Error task log:

2016-06-27 08:50:57.840 3980 ERROR rally.task.runner [-] Resource <Volume: 693905d5-b956-4fd1-9f05-32234216f0dd> has ERROR status.
Fault: n/a
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner Traceback (most recent call last):
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 66, in _run_scenario_once
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner deprecated_output = getattr(scenario_inst, method_name)(**kwargs)
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/cinder/volumes.py", line 215, in create_and_extend_volume
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner volume = self._create_volume(size, **kwargs)
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner f = func(self, *args, **kwargs)
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/cinder/utils.py", line 145, in _create_volume
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner check_interval=CONF.benchmark.cinder_volume_create_poll_interval
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/common/logging.py", line 236, in wrapper
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner return f(*args, **kwargs)
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 147, in wait_for
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner check_interval=check_interval)
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 211, in wait_for_status
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner resource = update_resource(resource)
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/utils.py", line 90, in _get_from_manager
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner fault=getattr(res, "fault", "n/a"))
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner GetResourceErrorStatus: Resource <Volume: 693905d5-b956-4fd1-9f05-32234216f0dd> has ERROR status.
2016-06-27 08:50:57.840 3980 ERROR rally.task.runner Fault: n/a

Steps to reproduce
==================
Install Rally on masternode and run task "create-and-extend-volume.yaml":

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start /opt/stack/rally-scenarios/cinder/create-and-extend-volume.yaml

Expected result
===============
Task run without errors

Actual result
=============
Errors described above in logs

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  199 compute nodes
  20 ceph nodes
  DVR enabled
  XVLAN
Image RC2 build:
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot for more detailed information:
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz

[Yuriy Nesenenko (ynesenenko)]

Cinder logs are missing in the diagnostic snapshot http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-28_13-13-50.tar.gz Without it it's hard to learn the cause of failing tests.

[Yuriy Nesenenko (ynesenenko)]

In attachment.zip cinder-volume.log is missing. Please add it.

[Denis Meltsaykin (dmeltsaykin)]

Closing as Invalid after a month in Incomplete.

[Andrew Kalach (akndex)]

*UPDATE*

The same error appeared during scale testing on MOS9.1

    CinderVolumes.create_and_attach_volume
    CinderVolumes.create_and_list_volume_backups
    GlanceImages.create_image_and_boot_instances
    NovaServers.boot_server

Full logs (diagnostic snapshot):
  http://mos-scale-share.mirantis.com/fuel-snapshot-2016-08-17_13-25-23.tar.gz

Controller logs (nova, rabbitmq, pacemaker, etc):
  http://mos-scale-share.mirantis.com/controller-logs-160818-e10.tar.gz

Controllers Cinder logs:
  http://mos-scale-share.mirantis.com/cinder-ctrl-logs-160818-e10.tar.gz

Controllers Glance logs:
  http://mos-scale-share.mirantis.com/glance-ctrl-logs-160818-e10.tar.gz

We tested on scale lab ENV-10
(contact me via email or slack for lab access details)