Cinder fails to attach second volume to Nova VM

[Valeriy Ponomaryov]

Cinder fails to attach second volume to Nova VM. This second volume gets "in-use" status, but does not have any attachments. Also, such volume cannot be detached from VM [4]. Test gerrit change [2] proves that commit to Cinder [3] is THE CAUSE of a bug.
Also, bug was reproduced even before merge of [3] with "gate-rally-dsvm-cinder" CI job [4], but, I assume, no one has paid attention to this.

Local testing shows that IF bug appears then volume never gets attached and list of attachments stays empty. And waiting between 'create' (wait until 'available' status) and 'attach' commands does not help at all.

How to reproduce:
1) Create VM
2) Create Volume
3) Attach volume (2) to the VM (1)
4) Create second volume
5) Try attach second volume (4) to VM (1) - it will fail.

[Tempest] Also, the fact that Cinder gates passed with [3] means that tempest does not have test that attaches more than one volume to one Nova VM. And it is also tempest bug, that should be addressed.

[Manila] In scope of Manila project, one of its drivers is broken - Generic driver that uses Cinder as backend.

[1] http://logs.openstack.org/64/386364/1/check/gate-manila-tempest-dsvm-postgres-generic-singlebackend-ubuntu-xenial-nv/eef11b0/logs/screen-m-shr.txt.gz?level=TRACE#_2016-10-14_15_15_19_898

[2] https://review.openstack.org/387915

[3] https://github.com/openstack/cinder/commit/6f174b412696bfa6262a5bea3ac42f45efbbe2ce ( https://review.openstack.org/385122 )

[4] http://logs.openstack.org/22/385122/1/check/gate-rally-dsvm-cinder/b0332e2/rally-plot/results.html.gz#/CinderVolumes.create_snapshot_and_attach_volume/failures

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/387560

[Valeriy Ponomaryov (vponomaryov)]

Problem appears when we get volume from Cinder with "in-use" status but with empty attachments.

[Valeriy Ponomaryov (vponomaryov)]

And Nova considers it as attached, but Cinder does not. And it is not attached indeed. First time problem was observed is 14th of October, Friday.

[Valeriy Ponomaryov (vponomaryov)]

It is not stable bug. It is concurrency-based bug. If we can do something in Manila then wait some time after each operation we do...

[Andrey Pavlov (apavlov-e)]

same behavior in ec2-api. Some attached volumes have empty attached list.

Request to cinder for attach volume - http://logs.openstack.org/48/386348/1/check/gate-functional-nova-network-dsvm-ec2api/226be9a/logs/screen-ec2-api.txt.gz#_2016-10-17_15_02_00_560

and then this volume has status 'in-use' and empty attachments list.

[Andrey Pavlov (apavlov-e)]

also in ec2-api logs I see the same behavior if instance is booted with additional block device mapping. This instance will have additional attachments but these volumes can have empty attachment list.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Valeriy Ponomaryov (<email address hidden>) on branch: master
Review: https://review.openstack.org/387560
Reason: Cannot be fixed in Manila. Should be fixed in Cinder/Nova.

[Valeriy Ponomaryov (vponomaryov)]

Andrey Pavlov,

Thanks! I confirm that problem is exactly with second+ volumes. Updated bug description.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/388654

[John Griffith (john-griffith)]

I'm not sure I follow here; this seems like something on the Manila side not Cinder or Nova. If I just create two volume (or 3 or 4) in Cinder and attach them to an instance that seems to work just fine:

jgriffith@bdr75:~/devstack$ nova volume-attach 15fb3379-1811-46ca-bdc3-35cccafb1233 1bf11fab-2329-4506-9cdb-3bf4af8688bc
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | 1bf11fab-2329-4506-9cdb-3bf4af8688bc |
| serverId | 15fb3379-1811-46ca-bdc3-35cccafb1233 |
| volumeId | 1bf11fab-2329-4506-9cdb-3bf4af8688bc |
+----------+--------------------------------------+
jgriffith@bdr75:~/devstack$ nova volume-attach 15fb3379-1811-46ca-bdc3-35cccafb1233 51dbb65b-5da8-435c-ae2d-895efac90dca

+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 51dbb65b-5da8-435c-ae2d-895efac90dca |
| serverId | 15fb3379-1811-46ca-bdc3-35cccafb1233 |
| volumeId | 51dbb65b-5da8-435c-ae2d-895efac90dca |
+----------+--------------------------------------+
jgriffith@bdr75:~/devstack$
jgriffith@bdr75:~/devstack$ nova volume-attach 15fb3379-1811-46ca-bdc3-35cccafb1233 a7c1c1eb-246b-4531-95fb-d6068ef3313e
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdd |
| id | a7c1c1eb-246b-4531-95fb-d6068ef3313e |
| serverId | 15fb3379-1811-46ca-bdc3-35cccafb1233 |
| volumeId | a7c1c1eb-246b-4531-95fb-d6068ef3313e |
+----------+--------------------------------------+
jgriffith@bdr75:~/devstack$ nova list; cinder list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| 15fb3379-1811-46ca-bdc3-35cccafb1233 | test | ACTIVE | - | Running | private=192.168.0.5 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
+--------------------------------------+--------+----------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+----------+------+-------------+----------+--------------------------------------+
| 1bf11fab-2329-4506-9cdb-3bf4af8688bc | in-use | - | 1 | lvmdriver-1 | false | 15fb3379-1811-46ca-bdc3-35cccafb1233 |
| 51dbb65b-5da8-435c-ae2d-895efac90dca | in-use | - | 1 | lvmdriver-1 | false | 15fb3379-1811-46ca-bdc3-35cccafb1233 |
| 6d4107b1-78a4-45e1-92c2-05182f4aab38 | in-use | fedora24 | 10 | solidfire | true | ...

[Andrey Pavlov (apavlov-e)]

@John, I saw such behavior in ec2-api project.
and I pasted link to logs of ec2-api in comment #5.
There is a link to line where ec2-api makes request to cinder for volume attach.
After this line we can see requests to cinder for volume information and response contains the attached volume but without attachment information.

[Valeriy Ponomaryov (vponomaryov)]

John,

1) Steps for reproduce include only Cinder and Nova, I find it *really* strange that you are talking about Manila problem.
2) Here is my list of actions reproducing bug using LATEST code of both - Cinder and Nova:
https://codepaste.net/gcjn9v#

So, I suspect, your environment does not satisfy criteria for the bug. Either Nova or Cinder have old code or you did not restart some services such as c-vol, but bug definitely exists!

I need to remind that only "master" branch has this bug and only with commit mentioned in bug report.

Question is not about "Whether this bug exists or not", question is about "HOW TO FIX IT?" only.

[Tomoki Sekiyama (tsekiyama)]

John,

I think this behaviour is introduced by your patch 6f174b41.
This overwrite the existing attachment entry in the db when the second volume is attached, as it is now skipping filtering by the volume id!

https://review.openstack.org/#/c/385122/

Valeriy, What happens if you revert the patch above?

[Andrey Pavlov (apavlov-e)]

@Tomoki, revert of patch above is here - https://review.openstack.org/#/c/388654/

ec2-api functional job is successful with reverted code.

[Tomoki Sekiyama (tsekiyama)]

@Andrey, thanks for the link. +1'ed.

[Valeriy Ponomaryov (vponomaryov)]

Tomoki,

you described bug completely right.

With problematic commit we start returning on line 945 [1], updating status but skipping creation of attachments for new volume. That is completely incorrect behavior.

[1] https://review.openstack.org/#/c/388654/1/cinder/volume/manager.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/389591

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/388654

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/389591
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=cf5c1fc360a55f4e736a095d81749ae8ec588dd2
Submitter: Jenkins
Branch: master

commit cf5c1fc360a55f4e736a095d81749ae8ec588dd2
Author: Zhiteng Huang <email address hidden>
Date: Fri Oct 21 01:12:25 2016 -0700

    Allow entry created in VolumeAttachment table for 2nd and later volumes

    In previous commit(https://github.com/openstack/cinder/commit/6f174b412696bfa6262a5bea3ac42f45efbbe2ce)
    there was a logic change to skip creating DB entry in
    VolumeAttachment table, this resulting one instance can only have
    one instance attaching to it.

    This change fix the bug and also make sure attach_volume() in
    volume manager always returns a DB record.

    Change-Id: Ia1d53f77a6d7fd987a871825f215ca296324119e
    Partial-bug: #1633535

[Ken'ichi Ohmichi (oomichi)]

I feel Cinder side bug seems to be fixed with https://review.openstack.org/389591 .
The commit message should be "Closes-Bug", not "Partial-Bug" I guess.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/401042

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/401042
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=09139283b8d67aadc79da8b625d00d08872e4270
Submitter: Jenkins
Branch: master

commit 09139283b8d67aadc79da8b625d00d08872e4270
Author: Ken'ichi Ohmichi <email address hidden>
Date: Thu Dec 1 14:36:22 2016 -0800

    Add a test for attaching 2 volumes to a server

    In Newton development, we could not attach multiple volumes to a
    single server. That seemed critical bug and it is good to block
    such changes in the future.

    Change-Id: Icdeafb5f30159205c89cb76a8d38399cefcb6127
    Closes-Bug: #1633535

[Andrey Volkov (avolkov)]

Can't reproduce behavior. See http://ix.io/nIs for details.

[Eric Harney (eharney)]

cf5c1fc3 Allow entry created in VolumeAttachment table for 2nd and later volumes

Should have been marked Closes-Bug: .