Incorrect status after volume attachment

[Kairat Kushaev]

The bug happened here during:
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/574/
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/579/
Please notice the bug is floating.

Scenario was the following:
            1. Create cluster with Neutron
            2. Add 3 nodes with controller role
            3. Add 3 nodes with compute and ceph-osd role
            4. Deploy the cluster
            5. Check ceph status
            6. Run OSTF tests
            7. Check the radosgw daemon is started

Expected result:
all tests passed.

Actual result:
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
2016-08-21 01:08:27 ERROR (nose_storage_plugin) fuel_health.tests.smoke.test_create_volume.VolumesTest.test_volume_create
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/smoke/test_create_volume.py", line 113, in test_volume_create
    volume, 'in-use')
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 182, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Step 7 failed: Attached volume status did not become "in-use".

Environment:
ubuntu xenial, jenkins-10.0.main.ubuntu.bvt_2-574.
See more details here:
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/574/injectedEnvVars/

[Kairat Kushaev (kkushaev)]

Cinder team,
could you please investigate why volume status wasn't "in-use"?

[Ivan Kolodyazhny (e0ne)]

According to Nova and Cinder logs, volume successfully attached. Looks like some env and/or timeout issue.

Roma, please, confirm it from Nova side

[Roman Podoliaka (rpodolyaka)]

Ivan, actually there is an error in Cinder logs:

http://paste.openstack.org/show/562222/

my understanding is that is the reason why OSTF stops waiting for a volume state to change.

It's not clear for me why pacemaker detected a split-brain of MySQL, though.

[Ivan Kolodyazhny (e0ne)]

According to logs, it was an issue with MySQL which is already fixed