3PAR: Failing tempest test_volume_create_get_update_delete_as_clone

[Ramy Asselin]

The following two tempest tests are failing on 3par iscsi:

14:52:47 {0} tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_as_clone [202.728027s] ... FAILED
14:56:31 {0} tempest.api.volume.test_volumes_get.VolumesGetTestXML.test_volume_create_get_update_delete_as_clone [201.540617s] ... FAILED

relevant log messages:
2014-07-28 22:56:20.889 18274 INFO cinder.volume.flows.manager.create_volume [req-d2cca136-40d4-4c03-8e45-fee712fe11b7 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Volume fe0a57b1-1a74-4f21-a921-07454e1e8661: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-fe0a57b1-1a74-4f21-a921-07454e1e8661'}
2014-07-28 22:56:21.563 18274 INFO cinder.volume.flows.manager.create_volume [req-d2cca136-40d4-4c03-8e45-fee712fe11b7 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Volume volume-fe0a57b1-1a74-4f21-a921-07454e1e8661 (fe0a57b1-1a74-4f21-a921-07454e1e8661): created successfully
2014-07-28 22:56:22.526 18274 INFO cinder.volume.flows.manager.create_volume [req-febd06b6-76d2-4e48-836b-39c7d9245c33 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Volume 61c7f33a-6a43-4091-8055-d8ff168d36d9: being created as source_vol with specification: {'status': u'creating', 'volume_size': 1, 'source_volstatus': u'available', 'source_volid': u'fe0a57b1-1a74-4f21-a921-07454e1e8661', 'volume_name': u'volume-61c7f33a-6a43-4091-8055-d8ff168d36d9'}
2014-07-28 22:56:23.427 18274 INFO cinder.volume.flows.manager.create_volume [req-febd06b6-76d2-4e48-836b-39c7d9245c33 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Volume volume-61c7f33a-6a43-4091-8055-d8ff168d36d9 (61c7f33a-6a43-4091-8055-d8ff168d36d9): created successfully
2014-07-28 22:56:24.441 18274 INFO cinder.volume.flows.manager.create_volume [req-d37cff0f-58c5-4569-bdf9-53372c89c892 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Volume b4665401-86e7-4297-9067-fd84d305bc57: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-b4665401-86e7-4297-9067-fd84d305bc57'}
2014-07-28 22:56:25.213 18274 INFO cinder.volume.flows.manager.create_volume [req-d37cff0f-58c5-4569-bdf9-53372c89c892 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Volume volume-b4665401-86e7-4297-9067-fd84d305bc57 (b4665401-86e7-4297-9067-fd84d305bc57): created successfully
2014-07-28 22:56:26.117 18274 INFO cinder.volume.manager [req-8bff77a4-3224-4931-9fde-c4776a49d508 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] volume b4665401-86e7-4297-9067-fd84d305bc57: deleting
2014-07-28 22:56:26.624 18274 INFO cinder.volume.manager [req-8bff77a4-3224-4931-9fde-c4776a49d508 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] volume b4665401-86e7-4297-9067-fd84d305bc57: deleted successfully
2014-07-28 22:56:26.654 18274 INFO cinder.volume.manager [req-8bff77a4-3224-4931-9fde-c4776a49d508 b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Updating volume status
2014-07-28 22:56:27.498 18274 INFO cinder.volume.manager [req-bfd35514-8494-40d6-bc81-019061bfab3c b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] volume 61c7f33a-6a43-4091-8055-d8ff168d36d9: deleting
2014-07-28 22:56:27.910 18274 ERROR cinder.volume.drivers.san.hp.hp_3par_common [req-bfd35514-8494-40d6-bc81-019061bfab3c b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Conflict (HTTP 409) 151 - invalid operation: Volume tuning is in progress
2014-07-28 22:56:27.910 18274 ERROR cinder.volume.drivers.san.hp.hp_3par_common [req-bfd35514-8494-40d6-bc81-019061bfab3c b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Conflict (HTTP 409) 151 - invalid operation: Volume tuning is in progress
2014-07-28 22:56:27.943 18274 ERROR cinder.volume.manager [req-bfd35514-8494-40d6-bc81-019061bfab3c b8a4f1d7ab1740cbb44ea4d1173c79b6 ca4738ac14184c1e8178f3d647c22940 - - -] Cannot delete volume 61c7f33a-6a43-4091-8055-d8ff168d36d9: volume is busy

Initial analysis:
2014-07-28 22:56:23.42 clone successfully created
2014-07-28 22:56:27.498 Delete
About 4 seconds

3PAR: 18665 online_vv_copy osv-YcfzOmpDQJGAVdj-Fo022Q done --- ---
2014-07-28 15:56:52 PDT start
2014-07-28 15:57:00 PDT done
About 8 seconds

3PAR is reporting the volume as available, and it is for most user operations (e.g. snapshot, attach, detach, etc.) Delete is an exception, where the user must wait for the clone to actually complete before the delete will return successfully.

[Anthony Lee (anthony-mic-lee)]

Currently waiting for a 3PAR firmware update to be released. Once that is released this bug can be fixed.

[Sean McGinnis (sean-mcginnis)]

Automatically unassigning due to inactivity.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/327840

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327840
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f0dda71610de8c48abb979c54a650e0b024134a6
Submitter: Jenkins
Branch: master

commit f0dda71610de8c48abb979c54a650e0b024134a6
Author: Alex O'Rourke <email address hidden>
Date: Thu Jun 9 09:13:13 2016 -0700

    3PAR: Fix delete volume when online clone

    When an online clone volume is deleted, the clone needs to be stopped
    first. On delete, a new exception is raised from the 3PAR. We need to
    check for the error code when hpeexceptions.HTTPConflict is raised.

    Change-Id: Iceb1b35e0312b109b2b19cd7d5c45b4a6242ccc5
    Closes-Bug: #1349639

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/333057

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex O'Rourke (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/327849
Reason: Please see https://review.openstack.org/#/c/333057/ for patch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/333057
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5248fca729f34d8b2dea780ce0d80c3d817992ad
Submitter: Jenkins
Branch: stable/mitaka

commit 5248fca729f34d8b2dea780ce0d80c3d817992ad
Author: Alex O'Rourke <email address hidden>
Date: Thu Jun 9 09:13:13 2016 -0700

    3PAR: Fix delete volume when online clone

    When an online clone volume is deleted, the clone needs to be stopped
    first. On delete, a new exception is raised from the 3PAR. We need to
    check for the error code when hpeexceptions.HTTPConflict is raised.

    Change-Id: Iceb1b35e0312b109b2b19cd7d5c45b4a6242ccc5
    Closes-Bug: #1349639
    (cherry picked from commit f0dda71610de8c48abb979c54a650e0b024134a6)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/cinder 9.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/cinder 8.1.0 release.