hitachi: one LDEV is assigned to multiple object
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
High
|
Unassigned |
Bug Description
Bug Summary:
If one LDEV is assigned to multiple objects(volume or snapshot), there is a risk of data loss or data corruption.
Occurrence conditions:
The conditions under which one LDEV is assigned to multiple objects are shown below.
(1). Object deletion failed with the following message:
... ERROR cinder.
(2). If (1) is occurred and another object is created, new LDEV with the same ID as the LDEV assigned to (1) is assigned. One LDEV is assigned to both objects: deleted in (1) and created in (2).
(3). If (2) is occurred and the delete command for the object that failed to be deleted in (1) is re-executed, the object and assigned LDEV are deleted, either. The deleted LDEV was also assigned to the object created in (2), so the data by the object created in (2) is lost.
(4). If a new object is created after data lost, the same LDEV ID as the LDEV assigned to the object created in (2) is assigned. The LDEV is also assigned to both objects: data lost in (3) and created in (4)
(5). If the delete command is executed for the object created in (2), the same failure as in (3) occurs.
How to check:
If count of ``provider_
Note that the higher the ``provider_
Use following command to know which ``provider_
$ podman exec galera-
output example of the command
$ podman exec galera-
+------
| provider_location | provider_
+------
| 2165 | 2 |
| 49001 | 3 |
| 2164 | 4 |
| 2163 | 5 |
| 27163 | 6 |
| 2160 | 8 |
| 2161 | 8 |
| 2162 | 8 |
+------
Recovery steps:
Overview of the steps:
1. If “provider_location” whose count is 2 or more, keep the latest object and destruct other objects.
2. Data in the LDEV which is assigned to destructed objects can’t recovery.
Details of recovery steps:
(1) finding “provider_location” whose count is 2 or more
run the command in the section "How to check"
(2) extracting objects information
Extract object ID and created date for each “provider_location” whose count is 2 or more found in step (1). The list is sorted in descending order, so keep the object which is at the top of the list(the latest) and others should be all destructed.
Command:
podman exec galera-
output example of the command:
$ podman exec galera-
+------
| object_type | id | created_at |
+------
| volume | ae023696-
| snapshot | 2fde155f-
| snapshot | 66aa1ee4-
| volume | b827bba1-
+------
$
In above example, the object whose id is ``ae023696-
(3) Destructing the objects
To destruct the objects which are extracted at step (2), set “1” to “delete” field in the DB.
Note:
Don’t use ``delete`` command in cinder, because ``delete`` command will delete LDEV, even other objects use it.
Following commands and examples destruct each volume(s) or snapshot(s):
(A) Destructing volume(s)
Commands:
cinder list | grep <volume id>
podman exec galera-
cinder list | grep <volume id>
Running the commands “cinder list | grep <volume id>” are to confirm the volume is destructed successfully.
output example of the commands:
$ cinder list | grep b827bba1-
| b827bba1-
$ podman exec galera-
$ cinder list | grep b827bba1-
$
(B) Destructing a snapshot(s)
Commands:
cinder snapshot-list | grep <snapshot id>
podman exec galera-
cinder snapshot-list | grep <snapshot id>
output example of the commands:
$ cinder snapshot-list | grep 66aa1ee4-
| 66aa1ee4-
$ podman exec galera-
$ cinder snapshot-list | grep 66aa1ee4-
$
Workaround:
If deleting volume or snapshot is failed with the message contains ``MSGID0731-E: Failed to communicate with the REST API server`` in the logs ``/var/
(1) Getting objects whose “provider_location” is same as the volume (or snapshot) which is failed in deleting
Command:
podman exec galera-
output example of the command:
$ podman exec galera-
+------
| object_type | id |
+------
| snapshot | 2fde155f-
| snapshot | 66aa1ee4-
| volume | ae023696-
| volume | b827bba1-
+------
(2) Deleting the object(s)
(a) when multiple objects are listed
Destruct objects along the steps in the section “Recovery steps”.
(b) when an object is listed
Delete the volume or the snapshot via Cinder. Reset the state of the volume or snapshot to “error” before deleting, because a volume or a snapshot in the state “error_deleting” can’t be deleted.
(b)-1 Deleting a volume
Commands:
cinder list | grep <volume>
cinder reset-state --state error <volume>
cinder list | grep <volume>
cinder delete <volume>
cinder list | grep <volume>
output example of the command:
$ cinder list | grep 7a6601f8-
| 7a6601f8-
$ cinder reset-state --state error 7a6601f8-
$ cinder list | grep 7a6601f8-
| 7a6601f8-
$ cinder delete 7a6601f8-
Request to delete volume 7a6601f8-
$ cinder list | grep 7a6601f8-
$
(b)-2 Deleting a snapshot
Commands:
cinder snapshot-list | grep <snapshot>
cinder snapshot-
cinder snapshot-list | grep <snapshot>
cinder snapshot-delete <snapshot>
cinder snapshot-list | grep <snapshot>
output example of the command:
$ cinder snapshot-list | grep c6898d6b-
| c6898d6b-
$ cinder snapshot-
$ cinder snapshot-list | grep c6898d6b-
| c6898d6b-
$ cinder snapshot-delete c6898d6b-
$ cinder snapshot-list | grep c6898d6b-
Fixing driver:
By driver fixing, when deleting an object which is created under the following conditions, check whether the LDEV is created when own object is created. If so, delete both the object and the LDEV. If not so, delete only the object.
- Volumes created by upstream drivers since the Wallaby release
- Snapshots created by the driver after applying the patch
The fixed driver will check that by comparing own object ID to be deleted and the object ID stored in LDEV nickname. The driver will store an object ID(*) in LDEV nickname when creating an object.
(*) Stored value is omitted "-" from the object ID, because maximum length of LDEV nickname is 32 digits characters.
Current upstream driver has already implemented to store object ID into LDEV nickname, when creating a volume. The fixed driver will be added features:
- Storing an object ID when creating a snapshot
- Comparing both object IDs when deleting an object.
On fixed driver and if LDEV nickname is not stored object ID(32digits hex), the driver will delete both the object and LDEV, because it avoid a risk for disk full.
Note:
- Do not change LDEV nickname
If user changed the nickname, driver could not compare correctly.
description: | updated |
description: | updated |
Changed in cinder: | |
importance: | Undecided → High |
Fix proposed to branch: master /review. opendev. org/c/openstack /cinder/ +/923618
Review: https:/