Volume fails to become 'available' after detach

[Leontii Istomin]

Corresponding upstream bug: https://bugs.launchpad.net/nova/+bug/1327218

api: '1.0'
astute_sha: 055b2d82fe8499b27c7047295e2e36a7a2c5d430
auth_required: true
build_id: 2015-04-16_21-30-10
build_number: '317'
feature_groups:
- mirantis
fuellib_sha: db5f39e96e7ab9f79691202755e547bf8242661f
fuelmain_sha: 0de2d2039e76839d339f977df45b111bef7200d6
nailgun_sha: 52d92c86e68602fb5dd2f3b8870a773d20a387ef
openstack_version: 2014.2-6.1
ostf_sha: b0991dbad159f53d335efa5d31cb94016ad5312e
production: docker
python-fuelclient_sha: 279ffc358e40dbdc162cfe76898cbd0874529f1f
release: '6.1'

Successfully deployed the following configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-gre,Ceph-all,Nova-debug,Nova-quotas, 6.1-317
Controllers:3 Computes:47

We performed

During rally create_snapshot_and_attach_volume scenario we have found the following error:
rally.log:
http://paste.openstack.org/show/229383/

request to detaching was sent to node-1:
messages:
http://paste.openstack.org/show/229384/

from nova-all.log:
http://paste.openstack.org/show/229385/

diagnostic snapshot is here: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-05-20_09-38-14.tar.xz

[Leontii Istomin (listomin)]

here is rally log of this scenario

[Roman Podoliaka (rpodolyaka)]

Looks like detach failed here: http://paste.openstack.org/show/230926/

(node-20, nova-compute.log)

[Roman Podoliaka (rpodolyaka)]

Related thread on openstack-dev ML - "[openstack-dev] [cinder][nova] condition needed to detach a volume"

[Roman Podoliaka (rpodolyaka)]

Based on the ML message, waiting for a volume to transit into 'in-use' state does not guarantee the volume can actually be detached immediately ("..Which means that the detach_volume is called before attach_volume functions actually ends and potentially update the connection_info field in the database....").

While this is a valid issue with unclear reporting of a volume state by Nova, this is not a blocker for us in 6.1 as this is only triggered by 'synthetic' tests.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Roman Podoliaka <email address hidden>
Review: https://review.fuel-infra.org/7575

[Eugene Nikanorov (enikanorov)]

Faced in with customer on 5.1 also.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Matt Riedemann <email address hidden>
Review: https://review.fuel-infra.org/7583

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/7575
Reason: Abandoned in favour of https://review.fuel-infra.org/#/c/7583/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/7583
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 80cb5ad5d63efb9a53f128deef6e5fbeebdb93ee
Author: Matt Riedemann <email address hidden>
Date: Sat Jun 6 05:53:28 2015

Save bdm.connection_info before calling volume_api.attach_volume

There is a race in attach/detach of a volume where the volume status
goes to 'in-use' before the bdm.connection_info data is stored in the
database. Since attach is a cast, the caller can see the volume go to
'in-use' and immediately try to detach the volume and blow up in the
compute manager because bdm.connection_info isn't set stored in the
database.

This fixes the issue by saving the connection_info immediately before
calling volume_api.attach_volume (which sets the volume status to
'in-use').

NOTE(mriedem): The block_device conflicts are due to using dot
notation when accessing object fields and in kilo the context is
no longer passed to bdm.save(). The test conflicts are due to moving
the test modules in kilo and passing the context on save().

Closes-Bug: #1457041

(cherry picked from commit bbf6348997fee02f9dadd556565f44005e2c7f23)
Change-Id: Ib95c8f7b66aca0c4ac7b92d140cbeb5e85c2717f

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-5.1/2014.1.1
Change author: Matt Riedemann <email address hidden>
Review: https://review.fuel-infra.org/7965

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Matt Riedemann <email address hidden>
Review: https://review.fuel-infra.org/8404

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/8404
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 80a0b7e5233ac75638dd49bfbb396bdb21a6aa21
Author: Matt Riedemann <email address hidden>
Date: Wed Jun 24 11:25:01 2015

Save bdm.connection_info before calling volume_api.attach_volume

There is a race in attach/detach of a volume where the volume status
goes to 'in-use' before the bdm.connection_info data is stored in the
database. Since attach is a cast, the caller can see the volume go to
'in-use' and immediately try to detach the volume and blow up in the
compute manager because bdm.connection_info isn't set stored in the
database.

This fixes the issue by saving the connection_info immediately before
calling volume_api.attach_volume (which sets the volume status to
'in-use').

NOTE(mriedem): The block_device conflicts are due to using dot
notation when accessing object fields and in kilo the context is
no longer passed to bdm.save(). The test conflicts are due to moving
the test modules in kilo and passing the context on save().

Closes-Bug: #1457041

(cherry picked from commit bbf6348997fee02f9dadd556565f44005e2c7f23)
Change-Id: Ib95c8f7b66aca0c4ac7b92d140cbeb5e85c2717f
(cherry picked from commit 80cb5ad5d63efb9a53f128deef6e5fbeebdb93ee)

[Miroslav Anashkin (manashkin)]

We've got customer who needs this fix backported to 5.1

[Eugene Nikanorov (enikanorov)]

https://review.fuel-infra.org/#/c/7965/1

[Alexander Gubanov (ogubanov)]

Verified on MOS 7.0 (build 257) - fixed !
Proof: http://pastebin.com/dhQb4Li8

[Sergii Rizvan (srizvan)]

On MOS 5.1.1 this bug is not reproducible after applying bugfix from https://bugs.launchpad.net/mos/+bug/1510957