Mirantis OpenStack

Cinder backup-restore operation is failing

Bug #1633534 reported by Paul Bessa
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Confirmed
Medium
Timur Sufiev
Mirantis OpenStack 10.0

Bug Description

Mirantis 9 environment.
Ubuntu 14.04
One Controller/Cinder Node
Two Compute Nodes

cinder backup operation is successful as listed below

cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+----------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+----------------+
| 7cce079c-47cf-4bcc-93f5-5a01626ff92c | 146bc76f-ff0e-49c8-82a6-b8566ffde704 | available | - | 5 | 104 | volumes_backup |
| 8d5b70c2-f31e-45b7-8734-b3d0cb133e00 | d969b910-e4be-41bf-a604-fb19f5bed757 | available | - | 2 | 42 | volumes_backup |
| e8b51e72-ae34-4555-9b56-c766f5f7812d | d969b910-e4be-41bf-a604-fb19f5bed757 | available | - | 2 | 42 | volumes_backup |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+----------------+

The cinder backup-restore operation fails with the following error message:

cinder backup-restore 7cce079c-47cf-4bcc-93f5-5a01626ff92c
ERROR: Invalid volume: Volume to be restored to must be available (HTTP 400) (Request-ID: req-5b0e3d51-998a-4e83-8d6a-4cfe62cd0d33)

Cinder list is listing the target volume as available.

# cinder list
+--------------------------------------+-----------+-----------------------------------------------------+------+-----------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------------------------------------------+------+-----------------+----------+--------------------------------------+
| 146bc76f-ff0e-49c8-82a6-b8566ffde704 | available | VMAX-v2-FC-vol-1 | 5 | vmax_1176_iscsi | false | |

Tags: area-cinder
Revision history for this message
Paul Bessa (paul-bessa) wrote :
Revision history for this message
Paul Bessa (paul-bessa) wrote :
Stanislaw Bogatkin (sbogatkin)
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
milestone: none → 9.2
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Cinder team, can you check this?

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → MOS Cinder (mos-cinder)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Paul, could you please share cinder-backup logs and 'cinder-mange service list' output?

Changed in fuel:
assignee: MOS Cinder (mos-cinder) → Paul Bessa (paul-bessa)
tags: added: area-cinder
Revision history for this message
Paul Bessa (paul-bessa) wrote :
Download full text (4.1 KiB)

Hello Ivan,

I reproduced the issue with clean/empty log files. There are two issues here. The 1st one...

There are two issues here...

The 1st issue is failing if the volume being backed up is attached.
cinder backup-create --force 2ec9d2f0-1d0b-402e-9e2a-4f2b4bed6b29

cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
 +--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
| 8ec37815-b3af-4a62-94fe-1708b6f02718 | 2ec9d2f0-1d0b-402e-9e2a-4f2b4bed6b29 | error | - | 10 | 0 | - |
 +--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
The backup is successful if the backup operation is performed to a volume that is detached.

 cinder backup-create --force 2ec9d2f0-1d0b-402e-9e2a-4f2b4bed6b29

cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+----------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
 +--------------------------------------+--------------------------------------+-----------+------+------+--------------+----------------+
| 8ec37815-b3af-4a62-94fe-1708b6f02718 | 2ec9d2f0-1d0b-402e-9e2a-4f2b4bed6b29 | error | - | 10 | 0 | - |
| c50ef076-b3f3-4214-9b6a-a166f71987c5 | 2ec9d2f0-1d0b-402e-9e2a-4f2b4bed6b29 | available | - | 10 | 206 | volumes_backup |
 +--------------------------------------+--------------------------------------+-----------+------+------+--------------+----------------+

The second issue is the failures with the volume restore operation -

cinder backup-restore c50ef076-b3f3-4214-9b6a-a166f71987c5
ERROR: Invalid volume: Volume to be restored to must be available (HTTP 400) (Request-ID: req-73c17d66-8481-431e-9107-e6a90ce037a1)

inder-manage service list
Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Binary Host Zone Status State Updated At RPC Version Object Version
cinder-backup node-2 nova enabled :-) 2016-11-02 20:59:38 2.0 1.3
cinder-volume node-2 nova enabled XXX 2016-10-31 22:22:16 2.0 1.3
cinder-scheduler node-2 nova enabled :-) 2016-11-02 20:59:35 2.0 1.3
cinder-volume node-2 nova enabled :-) 2016-11-02 20:59:38 2.0 1.3
cinder-volume node-2 nova enabled :-) 2016-11-02 20:59:33 2.0 1.3
cinder-volume n...

Read more...

Revision history for this message
Paul Bessa (paul-bessa) wrote :

Adding logs.

Revision history for this message
Paul Bessa (paul-bessa) wrote :

More logs...

Ivan Kolodyazhny (e0ne)
Changed in fuel:
assignee: Paul Bessa (paul-bessa) → Ivan Kolodyazhny (e0ne)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

As discussed with Paul, everything works as expected but we need to change 'migration policy' param manually.

Timur, for better UX please set 'migration policy' param value to 'on-demand' be default in Horizon for volume retype feature

Changed in fuel:
importance: High → Medium
affects: fuel → mos
Changed in mos:
milestone: 9.2 → none
assignee: Ivan Kolodyazhny (e0ne) → Timur Sufiev (tsufiev-x)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/28584

Changed in mos:
status: Confirmed → In Progress
Vitaly Sedelnik (vsedelnik)
Changed in mos:
milestone: none → 9.2
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/horizon (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/28584
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: b231e49444016e65029f4ae26fb1ad4d00859d8b
Author: Timur Sufiev <email address hidden>
Date: Wed Nov 30 14:23:10 2016

Change default migration policy when changing volume type

Change-Id: I4dae9c053c3c02c6dcddfedc20e76d35b3601c32
Closes-Bug: #1633534

Timur Sufiev (tsufiev-x)
Changed in mos:
status: In Progress → Fix Committed
Ekaterina Shutova (eshutova)
tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on snapshot 9.2 #593.
Cinder backup-restore works fine:
root@node-1:~# cinder backup-restore 101732d5-ecce-4c53-b938-f70110b3d56b
+-------------+-----------------------------------------------------+
| Property | Value |
 +-------------+-----------------------------------------------------+
| backup_id | 101732d5-ecce-4c53-b938-f70110b3d56b |
| volume_id | c595fedb-f777-49a0-9fbf-349bc897fbfc |
| volume_name | restore_backup_101732d5-ecce-4c53-b938-f70110b3d56b |
 +-------------+-----------------------------------------------------+
root@node-1:~# cinder list
+--------------------------------------+-----------+-----------------------------------------------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
 +--------------------------------------+-----------+-----------------------------------------------------+------+-------------+----------+-------------+
| a356116a-742d-433a-812f-76bdf1e51e20 | available | vol1 | 1 | volumes_lvm | true | |
| c595fedb-f777-49a0-9fbf-349bc897fbfc | available | restore_backup_101732d5-ecce-4c53-b938-f70110b3d56b | 1 | - | true | |
 +--------------------------------------+-----------+-----------------------------------------------------+------+-------------+----------+--

tags: removed: on-verification
Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (11.0/ocata)

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34186

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (mcp/newton)

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34226

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (mcp/ocata)

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34871

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/horizon (11.0/ocata)

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34186
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/horizon (mcp/ocata)

Reviewed: https://review.fuel-infra.org/34871
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 4ba922143512908b626f8469b0dc7af0ef3b4b0d
Author: Michael Dovgal <email address hidden>
Date: Wed Apr 26 16:05:25 2017

Change default migration policy when changing volume type

Change-Id: I4dae9c053c3c02c6dcddfedc20e76d35b3601c32
Closes-Bug: #1633534

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/horizon (mcp/newton)

Reviewed: https://review.fuel-infra.org/34226
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: 81bd714986940092bb4ca36b2c610faa2128a9c8
Author: Michael Dovgal <email address hidden>
Date: Mon May 8 12:37:32 2017

Change default migration policy when changing volume type

Change-Id: I4dae9c053c3c02c6dcddfedc20e76d35b3601c32
Closes-Bug: #1633534

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.