GPFS driver creates volume file on local disk when GPFS is unmounted

[Qin Zhao]

I enable GPFS driver on one cinder volume node. After I unmount GPFS from this node, the GPFS driver can continue to handle creating volume request, so that the volume file is created on local disk, instead of GPFS NSD.

[root@zhaoqin-RHEL-GPFS1 gpfs]# mmumount fs1
Sun Oct 20 11:35:53 CDT 2013: mmumount: Unmounting file systems ...
[root@zhaoqin-RHEL-GPFS1 gpfs]# mount | grep gpfs
[root@zhaoqin-RHEL-GPFS1 gpfs]#
[root@zhaoqin-RHEL-GPFS1 gpfs]# cinder create 1
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-10-20T16:36:11.236108 |
| display_description | None |
| display_name | None |
| id | 78906f80-4809-49e9-aceb-b21d6fdf226b |
| metadata | {} |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
[root@zhaoqin-RHEL-GPFS1 gpfs]# ls /gpfs/fs1
volume-78906f80-4809-49e9-aceb-b21d6fdf226b

Then, I attempt to remove this volume file using cinder command, but fails.

[root@zhaoqin-RHEL-GPFS1 gpfs]# cinder delete 78906f80-4809-49e9-aceb-b21d6fdf226b
[root@zhaoqin-RHEL-GPFS1 gpfs]# cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 78906f80-4809-49e9-aceb-b21d6fdf226b | error_deleting | None | 1 | None | false | |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

[Avishay Traeger (avishay-il)]

Delete fails because the driver runs 'mmclone show' on the file. The question is if its worth checking if the file system is mounted on every operation.

[Qin Zhao (zhaoqin)]

The current implementation of GPFS driver will check the GPFS mount path during driver initialization. However, when volume service is running, under some circumstances, the operator may also need to unmount GPFS without shutdown volume service. For instance, if the volume node is configured as a multi-backend node. When the operator need to unmount GPFS filesystem and perform some mantainence operation against NSD or something else, he would not hope to stop volume service, because the request to other backends still need to be handled. I feel it is better to have a way to let volume service temporarily refuse GPFS request. The current implementation will create file under a wrong place without any error notification. It will be hard for the operator to detect and recover this error.

[Bill Owen (billowen)]

There are 3 places where we need to add checks:
1. create_volume
2. copy_image_to_volume
3. _clone_image

In all other cases, if the filesystem is not mounted or gpfs is not running there will be an error when executing mmclone command, or the full path to the volume or snapshot is known, and will not be vaild.

I'll add a check in the 3 cases above to log an error and abort the operation if gpfs path is unavailable.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/54585

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54585
Committed: http://github.com/openstack/cinder/commit/cb1d1e4241c6ecb99797a46053c43b2e4b3b52e7
Submitter: Jenkins
Branch: master

commit cb1d1e4241c6ecb99797a46053c43b2e4b3b52e7
Author: Qin Zhao <email address hidden>
Date: Thu Oct 31 00:37:43 2013 +0800

    Examine if GPFS is mounted before writing data.

    If GPFS is not mounted, throw an exception and log the error message. That will
    prevent create_volume(), copy_image_to_volume() or _clone_image() to create
    file on local disk, which can not be removed by the driver code itself.

    Change-Id: If5d4c7a88e82362d2a87068202926b1ca82d9dff
    Closes-Bug: #1242511