Fail to delete volume snapshot created by GPFS driver

Bug #1242549 reported by Qin Zhao on 2013-10-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
High
Qin Zhao

Bug Description

I enable one cinder volume node with GPFS driver. Creating/deleting volume works correctly. However, deleting volume snapshot operation fails.

[root@zhaoqin-RHEL-GPFS1 install]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 88a10b05-78d9-495d-beb4-52863c016638 | available | zhaoqin-lvm | 1 | lvm | false | |
| d8a9cc4c-b0b8-481e-aa64-1af88bb3cf8b | available | zhaoqin-gpfs | 1 | gpfs | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
[root@zhaoqin-RHEL-GPFS1 install]# cinder snapshot-create --display_name=gpfs_snapshot d8a9cc4c-b0b8-481e-aa64-1af88bb3cf8b
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| created_at | 2013-10-21T07:20:36.284189 |
| display_description | None |
| display_name | gpfs_snapshot |
| id | 6274e3a0-24f0-47dc-914d-235a13047b6a |
| metadata | {} |
| size | 1 |
| status | creating |
| volume_id | d8a9cc4c-b0b8-481e-aa64-1af88bb3cf8b |
+---------------------+--------------------------------------+
[root@zhaoqin-RHEL-GPFS1 install]# cinder snapshot-list
+--------------------------------------+--------------------------------------+----------------+-------------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------------+-------------------+------+
| 6274e3a0-24f0-47dc-914d-235a13047b6a | d8a9cc4c-b0b8-481e-aa64-1af88bb3cf8b | available | gpfs_snapshot | 1 |
+--------------------------------------+--------------------------------------+----------------+-------------------+------+
[root@zhaoqin-RHEL-GPFS1 install]# cinder snapshot-delete 6274e3a0-24f0-47dc-914d-235a13047b6a
[root@zhaoqin-RHEL-GPFS1 install]# cinder snapshot-list
+--------------------------------------+--------------------------------------+----------------+-------------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------------+-------------------+------+
| 6274e3a0-24f0-47dc-914d-235a13047b6a | d8a9cc4c-b0b8-481e-aa64-1af88bb3cf8b | error_deleting | gpfs_snapshot | 1 |
+--------------------------------------+--------------------------------------+----------------+-------------------+------+

Qin Zhao (zhaoqin) wrote :

Notice one error in volume.log.

2013-10-21 02:21:23.049 21019 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/gpfs.py", line 400, in delete_snapshot
2013-10-21 02:21:23.049 21019 TRACE cinder.openstack.common.rpc.amqp os.rename(snapshot_path, snapshot_ts_path)
2013-10-21 02:21:23.049 21019 TRACE cinder.openstack.common.rpc.amqp OSError: [Errno 13] Permission denied
2013-10-21 02:21:23.049 21019 TRACE cinder.openstack.common.rpc.amqp

Avishay Traeger (avishay-il) wrote :

The rename command probably needs to be executed as root

tags: added: drivers gpfs
Changed in cinder:
importance: Undecided → High
Changed in cinder:
assignee: nobody → Bill Owen (billowen)

Fix proposed to branch: master
Review: https://review.openstack.org/52893

Changed in cinder:
assignee: Bill Owen (billowen) → Qin Zhao (zhaoqin)
status: New → In Progress
Bill Owen (billowen) wrote :

I think the issue is with the permissions defined on the gpfs file system. The cinder-volume process is running as cinder user; if the cinder user has permission to delete/rename files from $gpfs_mountpoint_base, then the operation will succeed.

Can you check how permissions are defined for the gpfs_mountpoint_base directory?

Qin Zhao (zhaoqin) wrote :

Hi Bill, my gpfs_mountpoint_base is /gpfs/fs1, which owned by root user with 755 permission code. That directory is created by GPFS program when I use mmcrfs to create the filesystem. I did not modify the permission of this folder.
Do you mean cinder user should be permitted to operation that gpfs directory?

Bill Owen (billowen) wrote :

Hi Qin Zhao,
In my testing I have volume directory (where gpfs_mountpoint_base points to) defined as owned by cinder user. With this setup, volume and snapshot deletes work.

That being said, I think your fix is a better way to do the rename, as it fits better with some changes planned for GPFS cinder driver to support configurations where gpfs client is not installed on node where cinder-volume is running.

Qin Zhao (zhaoqin) wrote :

Hi Bill,
If gpfs directory is owned by cinder user, that renaming should work. However, my understanding is cinder user just seem to be a user to run cinder program, so that it would not have the permission to operate GPFS without cinder program. Cinder code can specifically level up process prevelige temporarily to perform some operation, so that the security risk will be under controlled. If we authorize one user some extra permission, the seem be a security hole potentially.
I am glad to know this patch may help the scenario in which gpfs client is not installed on cinder volume node. I will consider to test that scenario to defect other potential problems. Thank you!

Reviewed: https://review.openstack.org/52893
Committed: http://github.com/openstack/cinder/commit/1212f66b188bbe40ac7219908eb47af26ebf2edc
Submitter: Jenkins
Branch: master

commit 1212f66b188bbe40ac7219908eb47af26ebf2edc
Author: chaochin <chaochin@zhaoqin-RHEL-GPFS1.(none)>
Date: Mon Oct 21 05:36:32 2013 -0500

    Let GPFS driver to rename snapshot with root permission

    Deleting GPFS volume snapshot operation fails, because cinder
    volume is not running with root permission.

    Change-Id: Id00357030da171fddbc6abf82603ceabe4db73ff
    Closes-Bug: #1242549

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-12-04
Changed in cinder:
milestone: none → icehouse-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in cinder:
milestone: icehouse-1 → 2014.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers