Too much volumes has an ERROR status

[Pavel]

Detailed bug description:
Problem discovered on MOS 9.1 ISO #495 RC2

Problem was discovered during execution of CinderVolumes.create_and_upload_volume_to_image. Cant access to the volumes. All volumes has an ERROR status

fuel/var/log/job-reports/rally.log.gz:1140574-2016-09-21 06:26:03.855 14007 DEBUG cinderclient.v2.client [-] RESP: [404] {'Content-Length': '109', 'X-Compute-Request-Id': 'req-6d825d5e-aabb-48b2-98ed-5830ca41640c', 'Vary': 'OpenStack-API-Version', 'Openstack-Api-Version': '2.0', 'Connection': 'close', 'Date': 'Wed, 21 Sep 2016 06:26:03 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'X-Openstack-Request-Id': 'req-6d825d5e-aabb-48b2-98ed-5830ca41640c'}
fuel/var/log/job-reports/rally.log.gz:1140575:RESP BODY: {"itemNotFound": {"message": "Volume 6ade1088-459e-4bfc-94e5-d9510de405b7 could not be found.", "code": 404}}

Steps to reproduce:
 Start rally test CinderVolumes.create_and_upload_volume_to_image.
Expected results:
 Test passed.
Actual result:
 Test failed.
Reproducibility:
 Iterations: 6, Failures: 6
Workaround:
 n/a
Impact:
 unknown
Description of the environment:
  * 10 virtual nodes:
     - CPU: 12 x 2.10 GHz
     - Disks: 2 drives (SSD - 80 GB, HDD - 931.5 GB), 1006.0 GB total
     - Memory: 2 x 16.0 GB, 32.0 GB total
     - NUMA topology: 1 NUMA node
  * Node roles:
    - 3 controllers (1 was is offline because of disk problems)
    - 7 computes
  * Details:
    - OS on controllers: Mitaka on Ubuntu 14.04
    - OS on computes: Mitaka on Ubuntu 14.04
    - Compute: KVM
    - Neutron with VLAN segmentation

Additional information:
 Diagnostic snapshot download link: http://mos-scale-share.mirantis.com/diagnostic_timmy_13env_05_10_2016/
Rally task results link:
 http://mos-scale.vm.mirantis.net/test_results/build_9.0.0-495/jenkins-13_env_run_rally-70-ubuntu-MSK-2016-09-20-22:23:43-2016-09-21-12:24:19/rally_report.html

[Denis Meltsaykin (dmeltsaykin)]

MOS Cinder team, could you please take a look at the diagnostic snapshot.

[Ivan Kolodyazhny (e0ne)]

Pavel, there are no cinder logs with volume in error state:(. Please, provide me access to the env once this issue will be reproduced again

[Michael Semenov (msemenov)]

Moved to Invalid.
Pavel, if you face this issue again, please file another bug in LP.