[10.0] Failed 3 OSTF tests in deployment of Ceph HA with RadosGW

[Kyrylo Romanenko]

Cluster deployed in following test:
        """Deploy ceph HA with RadosGW for objects
        Scenario:
            1. Create cluster with Neutron
            2. Add 3 nodes with controller role
            3. Add 3 nodes with compute and ceph-osd role
            4. Deploy the cluster
            5. Check ceph status
            6. Run OSTF tests
            7. Check the radosgw daemon is started
https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/tests/test_ceph.py#L462

Then failed 3 OSTF tests; should fail 0 tests. Names of failed tests:
  - Request volume list (failure) Time limit exceeded while waiting for volume listing to finish.
  - Launch instance with file injection (failure) Creating instance using the new security group has failed.
  - Launch instance, create snapshot, launch instance from snapshot (failure) Failed to get to expected status. In error state.

Failed job: https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1154/console

Both Cinder and Nova log containing many mentions of error message "WSREP has not yet prepared node for application use"

[Kyrylo Romanenko (kromanenko)]

Volume listing OSTF test failure:
(fuel_health.tests.sanity.test_sanity_compute.SanityComputeTest.test_list_volumes) Time limit exceeded while waiting for volume listing to finish. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/sanity/test_sanity_compute.py", line 58, in test_list_volumes
    self.volume_client)
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 180, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 1 failed: Time limit exceeded while waiting for volume listing to finish. Please refer to OpenStack logs for more details.
-------------------- >> begin captured logging << --------------------
fuel_health.common.test_mixins: INFO: STEP:1, verify action: 'volume listing'
fuel_health.common.test_mixins: INFO: Timeout 20s exceeded for volume listing
fuel_health.common.test_mixins: ERROR: Time limit exceeded while waiting for volume listing to finish.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 174, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 220, in __exit__
    raise AssertionError(msg)
AssertionError: Time limit exceeded while waiting for volume listing to finish.
--------------------- >> end captured logging << ---------------------

Errors in Cinder-volume log:
http://paste.openstack.org/show/597334/

[Kyrylo Romanenko (kromanenko)]

Errors on nova-conductor log looks similar: http://paste.openstack.org/show/597335/

ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1047, u'WSREP has not yet prepared node for application use')

[Kyrylo Romanenko (kromanenko)]

Also mysqld.log looks full of err output: http://paste.openstack.org/show/597336/