Cinder

3par iSCSI - revert-to-snap of a multi-attach volume leave volume in error state

Bug #1881730 reported by Tzach Shefi on 2020-06-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Undecided	Raghavendra Tilay

Bug Description

Description of problem: While testing revert-to-snapshot of a multi-attached 3par iSCSI backed volume, revert leaves volume in error state.
c-vol report 3par client exception:

2020-06-02 08:51:46.219 39 ERROR cinder.volume.manager raise exceptions.from_response(resp, body)
2020-06-02 08:51:46.219 39 ERROR cinder.volume.manager hpe3parclient.exceptions.HTTPForbidden: Forbidden (HTTP 403) 171 - Parent Volume is exported

BTW revert to snap of a simple 3par iSCSI volume worked fine.
It also worked fine for an LVM multi-attached volume.

Version-Release number of selected component (if applicable):

python3-cinderlib-1.0.1-0.20200310143137.199ebd4.el8ost.noarch
python3-cinderclient-5.0.1-0.20200326150227.8fa0882.el8ost.noarch
puppet-cinder-15.4.1-0.20200403212626.24b85a8.el8ost.noarch
openstack-cinder-15.1.1-0.20200403213514.cfa2d1b.el8ost.noarch
python3-cinder-15.1.1-0.20200403213514.cfa2d1b.el8ost.noarch

How reproducible:
2 times

Steps to Reproduce:
1. Boot up two instances

2. Create 3par mutli-attached volume
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type multiattach --name 3par-ma
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-06-02T08:41:17.000000 |
| description | None |
| encrypted | False |
| id | 225bc3df-93b3-41fe-b2a1-ee01296a6312 |
| metadata | {} |
| migration_status | None |
| multiattach | True |
| name | 3par-ma |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 20ece4159c8a4ea6b8cbb89edaafc920 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | a94a20fdd7534e45ba436d99995d9723 |
| volume_type | multiattach |
+--------------------------------+--------------------------------------+

3.
Attach to both instance, write data on volume

4. Create snapshot
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create 225bc3df-93b3-41fe-b2a1-ee01296a6312 --force --name spanMA
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2020-06-02T08:47:52.123503 |
| description | None |
| id | b999d61d-c5c0-4bdb-86f7-97370f28d271 |
| metadata | {} |
| name | spanMA |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | 225bc3df-93b3-41fe-b2a1-ee01296a6312 |
+-------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-show b999d61d-c5c0-4bdb-86f7-97370f28d271
+--------------------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------------------+--------------------------------------+
| created_at | 2020-06-02T08:47:52.000000 |
| description | None |
| id | b999d61d-c5c0-4bdb-86f7-97370f28d271 |
| metadata | {} |
| name | spanMA |
| os-extended-snapshot-attributes:progress | 100% |
| os-extended-snapshot-attributes:project_id | 20ece4159c8a4ea6b8cbb89edaafc920 |
| size | 1 |
| status | available |
| updated_at | 2020-06-02T08:47:52.000000 |
| volume_id | 225bc3df-93b3-41fe-b2a1-ee01296a6312 |
+--------------------------------------------+--------------------------------------+

5. Detach volume from both instances

(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst1 225bc3df-93b3-41fe-b2a1-ee01296a6312
(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst2 225bc3df-93b3-41fe-b2a1-ee01296a6312
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| 225bc3df-93b3-41fe-b2a1-ee01296a6312 | in-use | 3par-ma | 1 | multiattach | false | 87f36d4a-49a3-4ee8-88d3-f15b8cbf9b29 |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst1 225bc3df-93b3-41fe-b2a1-ee01296a6312
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| 225bc3df-93b3-41fe-b2a1-ee01296a6312 | available | 3par-ma | 1 | multiattach | false | |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+

Actual results:
Revert fails volume is left in error state.

Expected results:
Revert-to-snap should work leaving volume in available state.

Revision history for this message

Tzach Shefi (tshefi) wrote on 2020-06-02:

c-vol Edit (91.2 KiB, application/x-tar)

Revision history for this message

Raghavendra Tilay (raghavendrat) wrote on 2022-05-23:

output.txt Edit (13.0 KiB, text/plain)

Hi,

We tried to reproduce this using master (Zed).
This scenario isn't reproducible; it seems to be fixed.
Attaching cinder output (output.txt) and cinder logs (cinder-volume.log).

This bug was reported nearly two years ago.
It would be great if you can let us know if this bug is resolved now.

Regards,
Raghavendra Tilay.

Changed in cinder:
assignee:	nobody → Raghavendra Tilay (raghavendrat)

Revision history for this message

Raghavendra Tilay (raghavendrat) wrote on 2022-05-23:

cinder-volume.log Edit (579.9 KiB, text/plain)

Revision history for this message

Raghavendra Tilay (raghavendrat) wrote on 2022-05-23:

output.txt Edit (13.0 KiB, text/plain)

Revision history for this message

Raghavendra Tilay (raghavendrat) wrote on 2022-05-30:

Hi,

Waiting for response.

Revision history for this message

Raghavendra Tilay (raghavendrat) wrote on 2022-06-02:

Let me know if this bug can be closed.

Revision history for this message

Tzach Shefi (tshefi) wrote on 2022-06-06:

Download full text (18.1 KiB)

Hello,
Sorry for delay, resources were busy.

I've retested with:
python3-cinder-15.6.1-2.20220112174913.c093eda.el8ost.noarch
openstack-cinder-15.6.1-2.20220112174913.c093eda.el8ost.noarch
python3-cinderclient-5.0.2-2.20220107174846.7e9e31c.el8ost.noarch

My 3par needs a software update to reach 3.1.1 MU5,
see https://bugzilla.redhat.com/show_bug.cgi?id=2086763#c4
as I can't update my 3par yet, can't run the latest Cinder code.
Retesting with the latest version I can deploy/test.

Booted up two instances,
configured/created a MA/3par backed iSCSI volume

$ cinder type-create multiattach
+--------------------------------------+-------------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+-------------+-------------+-----------+
| f9fb9fd5-040e-444d-81f9-97d62d8da284 | multiattach | - | True |
+--------------------------------------+-------------+-------------+-----------+
$ cinder type-key multiattach set multiattach="<is> True"
$ cinder type-key multiattach set volume_backend_name=3par

Hello, 
Sorry for delay, resources were busy.

My 3par needs a software update to reach 3.1.1 MU5,
see https://bugzilla.redhat.com/show_bug.cgi?id=2086763#c4
as I can't update my 3par yet,  can't run the latest Cinder code.
Retesting with the latest version I can deploy/test.

Booted up two instances, 
configured/created a MA/3par backed iSCSI volume

$ cinder type-create multiattach
+--------------------------------------+-------------+-------------+-----------+
| ID                                   | Name        | Description | Is_Public |
+--------------------------------------+-------------+-------------+-----------+
| f9fb9fd5-040e-444d-81f9-97d62d8da284 | multiattach | -           | True      |
+--------------------------------------+-------------+-------------+-----------+
$ cinder type-key multiattach set multiattach="<is> True"
$ cinder type-key multiattach set volume_backend_name=3par

$ cinder create 2 --volume-type multiattach --name ma_vol1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2022-06-06T08:00:37.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | f95cbd68-2b25-4c6d-b642-017f74195de6 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | True                                 |
| name                           | ma_vol1                              |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 800f9746e3104fcfba11e6d3c94790c4     |
| replication_status             | None                                 |
| size                           | 2                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | d25961fc6a5740a9a189085c473cf153     |
| volume_type                    | multiattach                          |
+--------------------------------+--------------------------------------+

Create FS write data on volume:

(overcloud) [stack@undercloud-0 ~]$ ssh cirros@10.0.0.233
Warning: Permanently added '10.0.0.233' (ECDSA) to the list of known hosts.
$ sudo -i
# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    2G  0 disk 
# mkfs.ext4 /dev/vdb
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: 0c4ed1fb-723d-4f09-a89e-ec3f4dc40356
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

# mkdir ma

# mount /dev/vdb ma/
# touch ma/testfile.txt
# vi  ma/testfile.txt

# cat ma/testfile.txt 
kuku

Now lets attach same volume to inst2

Volume attached to both instance:
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+---------+------+-------------+----------+---------------------------------------------------------------------------+
| ID                                   | Status | Name    | Size | Volume Type | Bootable | Attached to                                                               |
+--------------------------------------+--------+---------+------+-------------+----------+---------------------------------------------------------------------------+
| f95cbd68-2b25-4c6d-b642-017f74195de6 | in-use | ma_vol1 | 2    | multiattach | false    | 14b7c1e3-ca98-4114-9d4b-89414fe93a25,04cfdbb1-7c8c-4741-9fdf-0fbc922561fd |
+--------------------------------------+--------+---------+------+-------------+----------+---------------------------------------------------------------------------+

Create a snapshot:
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create f95cbd68-2b25-4c6d-b642-017f74195de6 --force --name snapMA
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2022-06-06T08:08:34.698107           |
| description | None                                 |
| id          | c8af88f3-e895-46d8-bcdd-740e6afd0369 |
| metadata    | {}                                   |
| name        | snapMA                               |
| size        | 2                                    |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | f95cbd68-2b25-4c6d-b642-017f74195de6 |
+-------------+--------------------------------------+

Lets edit the file content:
# cat ma/testfile.txt 
kuku
More text added after snapshot was created.

Lets confirm we see the this change from the second instance:
(overcloud) [stack@undercloud-0 ~]$ ssh cirros@10.0.0.224
Warning: Permanently added '10.0.0.224' (ECDSA) to the list of known hosts.
$ sudo -i
# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    2G  0 disk 
# mkdir ma
# mount /dev/vdb ma/
# cat ma/testfile.txt 
kuku
More text added after snapshot was created.

Now lets detach from both instances:
(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst2 f95cbd68-2b25-4c6d-b642-017f74195de6
(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst1 f95cbd68-2b25-4c6d-b642-017f74195de6
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| ID                                   | Status    | Name    | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| f95cbd68-2b25-4c6d-b642-017f74195de6 | available | ma_vol1 | 2    | multiattach | false    |             |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+

Lets try to revert the snapshot:
cinder --os-volume-api-version=3.40 revert-to-snapshot

We still get the error:
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server [req-b8dad9e0-745a-4855-96f9-6853e81c5f45 d25961fc6a5740a9a189085c473cf153 800f9746e3104fcfba11e6d3c94790c4 - default default] Exception during message handling: hpe3parclient.exceptions.HTTPForbidden: Forbidden (HTTP 403) 171 - Parent Volume is exported
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 321, in _cs_request
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 297, in _time_request
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     resp, body = self.request(url, method, **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 262, in request
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server hpe3parclient.exceptions.HTTPForbidden: Forbidden (HTTP 403) 171 - Parent Volume is exported
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server 
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server 
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 1094, in revert_to_snapshot
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     LOG.exception(msg, msg_args)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     self.force_reraise()
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     raise value
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 1064, in revert_to_snapshot
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     self._revert_to_snapshot(context, volume, snapshot)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 1012, in _revert_to_snapshot
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     self.driver.revert_to_snapshot(context, volume, snapshot)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 784, in trace_logging_wrapper
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/hpe/hpe_3par_base.py", line 370, in revert_to_snapshot
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     common.revert_to_snapshot(volume, snapshot)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/hpe/hpe_3par_common.py", line 3644, in revert_to_snapshot
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     body = self.client.promoteVirtualCopy(snapshot_name, optional=optional)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/client.py", line 889, in promoteVirtualCopy
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     response, body = self.http.put('/volumes/%s' % snapshot, body=info)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 399, in put
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'PUT', **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 329, in _cs_request
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     resp, body = self._do_reauth(url, method, ex, **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 308, in _do_reauth
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 297, in _time_request
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     resp, body = self.request(url, method, **kwargs)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.6/site-packages/hpe3parclient/http.py", line 262, in request
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server hpe3parclient.exceptions.HTTPForbidden: Forbidden (HTTP 403) 171 - Parent Volume is exported
2022-06-06 08:20:40.215 26 ERROR oslo_messaging.rpc.server

Again as I said I didn't test latest code, 
as I'm using RH Openstack, can't deploy our latest release due to bz2086763.

I could retest with devstack, but if it worked for you,
there is no point in me retesting over latest devstack. 
Once I managed to update my 3par to MU5 I could retest on a more recent version of RH Openstack,
Which might also work as expected.

Revision history for this message

Raghavendra Tilay (raghavendrat) wrote on 2022-06-06:

Thanks Tzach for response.
Let us know your test result with newer 3par OS [3.3.1 MU5] and newer RH OpenStack [may be RHOSP17 or 18].
Will wait for feedback.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

redhat-bugs #2086763
[CLOSED WONTFIX] Edit

Bug watches keep track of this bug in other bug trackers.