Volume status changed from error to available

[Dudu granada]

Description
===========
After creating a volume which resulted with status = "error", an attempt to attach the volume to VM will fail and will change the volume status to - "available".

Steps to reproduce
==================
1. Create volume with status = "error".
2. Try to attach the volume to the VM.
3. Check the volume status.

Expected result
===============
1. Volume will be created with status = "error".
2. Attachment to VM will failed.
3. Volume status will be - "error"

Actual result
=============
1. Volume created in status = "error".
2. Attachment attempt to VM will fail.
3. Volume status changed from "error" to "available".

Environment
===========
[stack@undercloud~(tm5)]$ openstack --version
openstack 3.14.2

[stack@undercloud~(tm5)]$ rpm -qa --last | grep nova
openstack-nova-scheduler-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:21 PM UTC
openstack-nova-placement-api-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:21 PM UTC
openstack-nova-conductor-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:21 PM UTC
openstack-nova-api-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:21 PM UTC
openstack-nova-compute-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:20 PM UTC
python-nova-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:18 PM UTC
openstack-nova-common-17.0.6-0.20180704141006.3163c93.el7.noarch Mon 09 Jul 2018 07:38:18 PM UTC
python2-novaclient-10.1.0-0.20180605113302.ed73aad.el7.noarch Mon 09 Jul 2018 07:37:49 PM UTC
puppet-nova-12.4.1-0.20180617130812.b5284f1.el7.noarch Mon 09 Jul 2018 07:33:17 PM UTC
[stack@undercloud~(tm5)]$
[stack@undercloud~(tm5)]$ rpm -qa --last | grep cinder
python2-cinderclient-3.5.0-0.20180211213738.1de605c.el7.centos.noarch Mon 09 Jul 2018 07:37:49 PM UTC
puppet-cinder-12.4.1-0.20180628102250.641e036.el7.noarch Mon 09 Jul 2018 07:33:17 PM UTC
[stack@undercloud~(tm5)]$

[stack@undercloud~(tm5)]$ sudo dmidecode | egrep -i 'manufacturer|product|vendor'
        Vendor: Seabios
        Manufacturer: Red Hat
        Product Name: KVM
        Manufacturer: Bochs

Storage:
ceph version 12.2.4

[Matt Riedemann (mriedem)]

OK so nova and cinder are both at the queens release, which means when you attach the volume to the server, the compute API should create a volume attachment record on the given volume. If the volume is in 'error' status I'd expect that to fail like how the old volume reserve action would fail for a volume in error status.

Looking at the cinder volume attachment create code, it should fail if the volume isn't in 'available' or 'downloading' status:

https://github.com/openstack/cinder/blob/9bc9a528ef46522fd727a2a047da435c658a15c5/cinder/volume/api.py#L2084

It seems that this is the problem - because there is an instance_uuid, override=True:

https://github.com/openstack/cinder/blob/9bc9a528ef46522fd727a2a047da435c658a15c5/cinder/volume/api.py#L2093

And then the volume statis us changed to reserved:

https://github.com/openstack/cinder/blob/9bc9a528ef46522fd727a2a047da435c658a15c5/cinder/volume/api.py#L2111

Then once the attach on the compute actually fails, we'd delete the attachment from the nova side and the volume will be back in 'available' status because it no longer has any attachments:

https://github.com/openstack/cinder/blob/9bc9a528ef46522fd727a2a047da435c658a15c5/cinder/volume/api.py#L2193

Compare that to how the volume "reserve" action works:

https://github.com/openstack/cinder/blob/9bc9a528ef46522fd727a2a047da435c658a15c5/cinder/volume/api.py#L690

If the volume isn't 'available' it will fail.

[Matt Riedemann (mriedem)]

Furthermore we won't get here if:

https://github.com/openstack/cinder/blob/9bc9a528ef46522fd727a2a047da435c658a15c5/cinder/volume/api.py#L2100

If the volume doesn't have any attachments. Looks like that for loop should have an else clause to set override back to False.

This is a cinder bug so I've changed the project.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/590037

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/590037
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9c0123eb70b0928786d81469cc3ccbd4f30f38fe
Submitter: Zuul
Branch: master

commit 9c0123eb70b0928786d81469cc3ccbd4f30f38fe
Author: Matt Riedemann <email address hidden>
Date: Wed Aug 8 16:44:33 2018 -0400

    Fix _attachment_reserve to not allow attaching an invalid status volume

    It is currently possible to create a volume attachment for a server
    when the volume is in error status because of the override logic
    in the _attachment_reserve method. What results is that the volume
    attach operation fails in nova-compute which rolls back and deletes
    the volume attachment, which puts the volume into 'available' status
    because it no longer has any attachments, which in fact it should
    have never allowed the attachment create/reserve in the first place.

    This updates the override logic such that a volume without any
    attachments which is in an invalid status will result in an error
    being raised.

    Change-Id: Id9cf2f510684cd296ffbcaf53d11889cfe8973b9
    Closes-Bug: #1785050

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/590327

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/590327
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=1c06339d6a805146e777cdba8ef550f14bada9b2
Submitter: Zuul
Branch: stable/queens

commit 1c06339d6a805146e777cdba8ef550f14bada9b2
Author: Matt Riedemann <email address hidden>
Date: Wed Aug 8 16:44:33 2018 -0400

    Fix _attachment_reserve to not allow attaching an invalid status volume

    It is currently possible to create a volume attachment for a server
    when the volume is in error status because of the override logic
    in the _attachment_reserve method. What results is that the volume
    attach operation fails in nova-compute which rolls back and deletes
    the volume attachment, which puts the volume into 'available' status
    because it no longer has any attachments, which in fact it should
    have never allowed the attachment create/reserve in the first place.

    This updates the override logic such that a volume without any
    attachments which is in an invalid status will result in an error
    being raised.

    Change-Id: Id9cf2f510684cd296ffbcaf53d11889cfe8973b9
    Closes-Bug: #1785050
    (cherry picked from commit 9c0123eb70b0928786d81469cc3ccbd4f30f38fe)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/592212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/592212
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=aae373d7d58b8c964d30b3cd709fe03fee15195a
Submitter: Zuul
Branch: stable/pike

commit aae373d7d58b8c964d30b3cd709fe03fee15195a
Author: Matt Riedemann <email address hidden>
Date: Wed Aug 8 16:44:33 2018 -0400

    Fix _attachment_reserve to not allow attaching an invalid status volume

    It is currently possible to create a volume attachment for a server
    when the volume is in error status because of the override logic
    in the _attachment_reserve method. What results is that the volume
    attach operation fails in nova-compute which rolls back and deletes
    the volume attachment, which puts the volume into 'available' status
    because it no longer has any attachments, which in fact it should
    have never allowed the attachment create/reserve in the first place.

    This updates the override logic such that a volume without any
    attachments which is in an invalid status will result in an error
    being raised.

    Conflicts:
        cinder/tests/unit/volume/test_volume.py
        cinder/volume/api.py

    Change-Id: Id9cf2f510684cd296ffbcaf53d11889cfe8973b9
    Closes-Bug: #1785050
    (cherry picked from commit 9c0123eb70b0928786d81469cc3ccbd4f30f38fe)
    (cherry picked from commit 1c06339d6a805146e777cdba8ef550f14bada9b2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.2.0 release.