cinder backup service should change backup & volume status in absence of swift service

[satyadev svn]

stopped one of swift service [s-object]
Take back up of any existing volume
observed exception in c-bak service logs however status of backup remains in 'creating' only

commands :
stop s-object service

ssatya@autojuno:~/juno/devstack$ cinder backup-create b7d8b000-d69c-4411-ab00-a3911c64ecdc --name test_backup2
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | c19ecea3-a57c-49b4-94ed-22854a031f53 |
| name | test_backup2 |
| volume_id | b7d8b000-d69c-4411-ab00-a3911c64ecdc |
+-----------+--------------------------------------+
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | None |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ cinder list
+--------------------------------------+------------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+------------+------+------+-------------+----------+-------------+
| 74c5a098-b2a8-4e80-b82d-18441396280e | available | test | 1 | None | false | |
| b7d8b000-d69c-4411-ab00-a3911c64ecdc | backing-up | test | 1 | None | false | |
+--------------------------------------+------------+------+------+-------------+----------+-------------+
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 74c5a098-b2a8-4e80-b82d-18441396280e | available | test | 1 | None | false | |
| b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test | 1 | None | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ date
Wed Oct 1 13:50:27 IST 2014
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
ssatya@autojuno:~/juno/devstack$ date
Wed Oct 1 13:55:27 IST 2014
ssatya@autojuno:~/juno/devstack$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+
| c19ecea3-a57c-49b4-94ed-22854a031f53 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | creating | test_backup2 | 1 | None | volumebackups |
| df49c43a-0ee8-43ad-8254-5c8526922d73 | b7d8b000-d69c-4411-ab00-a3911c64ecdc | available | test_backup | 1 | 2 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+--------------+------+--------------+---------------+

[John Griffith (john-griffith)]

Wow.. this just keeps getting worse :(

[Duncan Thomas (duncan-thomas)]

Not sure that this is a bug. Backup to swift requires a working swift installation - the redundancy level of yourswift depends on how you're configured it.

Expecting cinder backup to deal with any fails in swift other than the most gross type is out of scope.

[John Griffith (john-griffith)]

Most certainly a bug as far as I'm concerned. No reason to sit at "creating" forever... just detect that swift isn't available, timeout or whatever and change the status to failed/error. Sitting in an active state forever is not acceptable IMO.

[Duncan Thomas (duncan-thomas)]

Fair point, we should time out and go to error

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/127261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127261
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=82716b4ac836024b968c76db75be8a92ede0e226
Submitter: Jenkins
Branch: master

commit 82716b4ac836024b968c76db75be8a92ede0e226
Author: Vipin Balachandran <email address hidden>
Date: Thu Oct 9 19:35:46 2014 +0530

    Truncate fail_reason to column length

    During create_backup failure handling, backup_update fails with
    DataError ("Data too long for column") if the fail_reason is
    greater than 255 characters. As a result, backup status is stuck
    in 'creating' state. This patch avoids the problem by truncating
    fail_reason to 255 characters before update.

    Closes-Bug: #1376199
    Change-Id: If0d616b81d3869f7ea110caab8cf4140cf5c5c9e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/130211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130211
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c007350c500d31a38f709f8aae719dce067ebc65
Submitter: Jenkins
Branch: stable/juno

commit c007350c500d31a38f709f8aae719dce067ebc65
Author: Vipin Balachandran <email address hidden>
Date: Thu Oct 9 19:35:46 2014 +0530

    Truncate fail_reason to column length

    During create_backup failure handling, backup_update fails with
    DataError ("Data too long for column") if the fail_reason is
    greater than 255 characters. As a result, backup status is stuck
    in 'creating' state. This patch avoids the problem by truncating
    fail_reason to 255 characters before update.

    Closes-Bug: #1376199
    Change-Id: If0d616b81d3869f7ea110caab8cf4140cf5c5c9e
    (cherry picked from commit 82716b4ac836024b968c76db75be8a92ede0e226)