Volumes are stuck in creating state after reboot primary controller

[Andrey Sledzinskiy]

{

    "build_id": "2014-08-11_12-45-06",
    "mirantis": "yes",
    "build_number": "169",
    "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
    "nailgun_sha": "04ada3cd7ef14f6741a05fd5d6690260f9198095",
    "production": "docker",
    "api": "1.0",
    "fuelmain_sha": "43374c706b4fdce28aeb4ef11e69a53f41646740",
    "astute_sha": "6db5f5031b74e67b92fcac1f7998eaa296d68025",
    "release": "5.0.1",
    "fuellib_sha": "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"

}

Steps:
1. Create next cluster - Centos, HA, Neutron GRE, Cinder for volumes, 3 controllers, 1 compute, 1 cinder
2. Deploy cluster
3. After successfull deployment make reboot of primary controller
4. Wait for half an hour
5. Try to create volume

Expected - volume can be created
Actual - volume stuck in creation state

Errors in cinder-volume.log on cinder node (node-15):
oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'cinder-scheduler': (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'

Errors in nova-compute.log on compute node (node-14):
oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'conductor': [Errno 32] Broken pipe

Logs are attached

Logs are attached

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20455
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.1-stable-20455/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20456
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20456/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20456
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20456/ubuntu

[Meg McRoberts (dreidellhasa)]

For the Release Notes, what do they do if this happens?

[Ivan Kolodyazhny (e0ne)]

Here is patch to fix this problem: http://paste.openstack.org/show/95051/

[Dmitry Borodaenko (angdraug)]

Workaround is to restart cinder services, force-delete and re-create the affected volumes.

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20456
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20456
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu

[Dmitry Mescheryakov (dmitrymex)]

Please port this change to MOS 5.0.2

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20681
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-20681/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20681
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.2-stable-20681/ubuntu

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20681
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: http://gerrit.mirantis.com/20681
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.2-stable/ubuntu

[Tatyanka (tatyana-leontovich)]

Set to critical according to fail a lot of cases of patching feature

[Tatyanka (tatyana-leontovich)]

Still reproduced on {"build_id": "2014-08-20_02-01-17", "ostf_sha": "c6ecd0137b5d7c1576fa65baef0fc70f9a150daa", "build_number": "459", "auth_required": true, "api": "1.0", "nailgun_sha": "36d27ff737b361f92093986d061bbfc1670bee45", "production": "docker", "fuelmain_sha": "365fc0bfe9d5e4ce38101d9158f66347bf32c310", "astute_sha": "efe3cb3668b9079e68fb1534fd4649ac45a344e1", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "fa23adb05c58fdad5011a3ad806467eb3d883217"} Package cinder has changes

http://paste.openstack.org/show/97798/

[Dmitry Mescheryakov (dmitrymex)]

Clarification from chat with Tatyana: the issue was reproduced on 5.0.2

[Ivan Kolodyazhny (e0ne)]

Works on ISO:
{
build_id: "2014-08-22_14-59-17",
ostf_sha: "907f25f8fad39b177bf6a66fba9785afa7dd8008",
build_number: "469",
auth_required: true,
api: "1.0",
nailgun_sha: "e9dd053fce12908b5a5a404435ce2e8200450a23",
production: "docker",
fuelmain_sha: "5a7df58786db7962a1774b5be3611c4e7543015d",
astute_sha: "ac520b09525af4551e730b1c1f78170fefaf3cb8",
feature_groups: [
"mirantis"
],
release: "5.1",
fuellib_sha: "bddba1e854a6b0350e844a0baad50816d3cc8e28"
}

Unable to reproduce an issue with provided steps

[Tatyanka (tatyana-leontovich)]

Ivan you closes this one issue https://bugs.launchpad.net/fuel/+bug/1356942 as duplicate of this , could you confirm that it is worked too?

[Ivan Kolodyazhny (e0ne)]

Andrey, please, confirm bug as fixes with the latest oslo.messaging lib or provide more detailed steps to reproduce

[Dmitry Mescheryakov (dmitrymex)]

Andrey, for your reference, the fix for oslo.messaging will appear in 5.0.2 only after https://bugs.launchpad.net/mos/+bug/1359705 is fixed

[Dmitry Mescheryakov (dmitrymex)]

The fix to bug 1359705 will be in ISO starting from build #487. Andrey, please reverify bug on that ISO.

[Dmitry Mescheryakov (dmitrymex)]

Also the bug was wrongly moved to incomplete in release 5.1. Actually the fix is committed there and no problems were observed so far.

[Dmitry Mescheryakov (dmitrymex)]

Move to 'fix committed' to indicate readiness to be tested by QA folks, as suggested by Tatyana.

[Dmitry Mescheryakov (dmitrymex)]

The #487 does not contain the fix. I misunderstood 5.1 upgrade tarball build sequence. I will post here 5.1 build number which includes the fix once it is made.