Could not create volume after restart environment ceph ha

[Egor Kotko]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "490"
  build_id: "2014-08-31_00-01-17"
  astute_sha: "bc60b7d027ab244039f48c505ac52ab8eb0a990c"
  fuellib_sha: "2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f"
  ostf_sha: "4dcd99cc4bfa19f52d4b87ed321eb84ff03844da"
  nailgun_sha: "d25ed02948a8be773e2bd87cfe583ef7be866bb2"
  fuelmain_sha: "109812be3425408dd7be192b5debf109cb1edd4c"

System test: "ceph_ha_restart" was fail on OSTF (Create volume and attach it to instance (fuel_health.tests.smoke.test_create_volume.VolumesTest.test_volume_create). Test status: failure, message: Volume was not created.)
Log (cinder-api.log on node-1) contains:
http://paste.openstack.org/show/104156/

http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.thread_3/153/testReport/%28root%29/ceph_ha_restart/ceph_ha_restart/

[Andrew Woodward (xarses)]

traceback shows rabbit connection errors, removing ceph tag

[Dennis Dmitriev (ddmitriev)]

In the test "ceph_ha_restart" volumes are not created because of cinder-scheduler and cinder-volume services are not started. Cinder services depend on rabbitmq functionality and become available only after 'rabbitmq' cluster start.

It could be checked with "cinder service-list" command on any controller of the cluster.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123014

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/123015

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123014
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=27908dd829fcaf7d3395f3da3c5a42263a629532
Submitter: Jenkins
Branch: master

commit 27908dd829fcaf7d3395f3da3c5a42263a629532
Author: Dennis Dmitriev <email address hidden>
Date: Sun Sep 21 22:15:17 2014 +0300

    Added check for Cinder services

    After cluster nodes restart, volumes creation are failed
    for several minutes because cinder services are down
    due to rabbitmq is not ready yet. We need wait until
    cinder-scheduler and cinder-volumes are up on controllers
    before doing OSTF tests.

    Change-Id: Ie9c35368e0f5e399ee2fe69c7132a4232fe40690
    Closes-Bug: #1363994

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123015
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=396210066bdfbaa21697e516a8c63ced6493bc5d
Submitter: Jenkins
Branch: stable/5.1

commit 396210066bdfbaa21697e516a8c63ced6493bc5d
Author: Dennis Dmitriev <email address hidden>
Date: Sun Sep 21 22:15:17 2014 +0300

    Added check for Cinder services

    After cluster nodes restart, volumes creation are failed
    for several minutes because cinder services are down
    due to rabbitmq is not ready yet. We need wait until
    cinder-scheduler and cinder-volumes are up on controllers
    before doing OSTF tests.

    Change-Id: Ie9c35368e0f5e399ee2fe69c7132a4232fe40690
    Closes-Bug: #1363994

[Dmitry Borodaenko (angdraug)]

Why is this back to In Progress?

[Dennis Dmitriev (ddmitriev)]

It was mistake.