Cinder backup service: Volume backup of 50GB failed

[Kyrylo Romanenko]

Volume backup failed once while attempting to backup 50GB volume.
Steps:
1.
# cinder create 50 --display-name VOLUME50
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2015-04-21T07:42:19.029018 |
| display_description | None |
| display_name | VOLUME50 |
| encrypted | False |
| id | de682695-c39e-4bf6-981e-d9a7d227ba5f |
| metadata | {} |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+

2.
# cinder backup-create VOLUME50
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | a90a2424-94ae-4cc1-acff-39e30c4d2af5 |
| name | None |
| volume_id | de682695-c39e-4bf6-981e-d9a7d227ba5f |
+-----------+--------------------------------------+

Then wait until backup has failed.
3.
# cinder backup-show a90a2424-94ae-4cc1-acff-39e30c4d2af5
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| availability_zone | None |
| container | volumebackups |
| created_at | 2015-04-21T07:42:38.000000 |
| description | None |
| fail_reason | Object PUT failed: http://172.16.57.35:8080/v1/AUTH_8bfc15da1a1145dd8dceae51986e9f47/volumebackups/volume_de682695-c39e-4bf6-981e-d9a7d227ba5f/20150421074252/az_nova_backup_a90a2424-94ae-4cc1-acff-39e30c4d2af5-01006 401 Unauthorized [first 60 chars of re |
| id | a90a2424-94ae-4cc1-acff-39e30c4d2af5 |
| name | None |
| object_count | None |
| size | 50 |
| status | error |
| volume_id | de682695-c39e-4bf6-981e-d9a7d227ba5f |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Backup service log attached. There are many error messages there.

Version:
kromanenko@kromanenko-pc:~$ ssh root@172.16.57.34
root@172.16.57.34's password:
Last login: Tue Apr 21 07:05:07 2015 from 172.18.196.49
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2-6.1"
  api: "1.0"
  build_number: "317"
  build_id: "2015-04-16_21-30-10"
  nailgun_sha: "52d92c86e68602fb5dd2f3b8870a773d20a387ef"
  python-fuelclient_sha: "279ffc358e40dbdc162cfe76898cbd0874529f1f"
  astute_sha: "055b2d82fe8499b27c7047295e2e36a7a2c5d430"
  fuellib_sha: "db5f39e96e7ab9f79691202755e547bf8242661f"
  ostf_sha: "b0991dbad159f53d335efa5d31cb94016ad5312e"
  fuelmain_sha: "0de2d2039e76839d339f977df45b111bef7200d6"

Centos 6.5, CentOS default kernel, Neutron VLAN, QEMU, Ceph RDB for Cinder and Glance, Replication Factor = 2.
3 nodes:
1 Controller + Ceph OSD
1 Compute + Ceph OSD
1 Ceph OSD

[Kyrylo Romanenko (kromanenko)]

Diagnostic snapshot: https://drive.google.com/file/d/0B6E70aHvCcRQVS1TZGNmdGxycnM/view?usp=sharing

[Kyrylo Romanenko (kromanenko)]

Reproduced this bug one more time on another environment. This time backup of 60 GB volume failed.

Ubuntu 14.04.1 , Neutron VLAN, QEMU, Ceph RDB for Cinder and Glance, Replication Factor = 1.

3 nodes:
1 Controller
1 Compute
1 Ceph OSD

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2-6.1"
  api: "1.0"
  build_number: "329"
  build_id: "2015-04-20_22-54-31"
  nailgun_sha: "69fca3bada96a48ab19b969621e5128390b7052e"
  python-fuelclient_sha: "b4f1ddffd5263489090b65e662173e9e11aafd94"
  astute_sha: "bf1751a4fe0d912325e3b4af629126a59c1b2b51"
  fuellib_sha: "b878f1ae8e1d4c6f2f463e810b1f8014df5b22a5"
  ostf_sha: "df8db1f48f03b18126ce5ec65317a1eb83a5a95f"
  fuelmain_sha: "0c0e299dcc0276bb8262256148c6ee975857ee86"

[Kyrylo Romanenko (kromanenko)]

Diagnostic snapshot
https://drive.google.com/file/d/0B6E70aHvCcRQY1Y3d3JOaEJVemc/view?usp=sharing

[Stanislav Makar (smakar)]

storage:
  iser: false
  volumes_ceph: true
  objects_ceph: false
  ephemeral_ceph: false
  volumes_lvm: false
  images_vcenter: false
  osd_pool_size: '2'
  pg_num: 512
  images_ceph: true
  metadata:
    weight: 60
    label: Storage

From logs
As we see Ceph is enabled only for volumes

grep backup_driver node-1.test.domain.local/etc/cinder/*
node-1.test.domain.local/etc/cinder/cinder.conf:#backup_driver=cinder.backup.drivers.swift

no backup_driver
so that means that cinder backup not vonfigured correctly
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/4/html/Configuration_Reference_Guide/section_backup-drivers.html

[Kyrylo Romanenko (kromanenko)]

Interesting catch, it looks to be a bug, because i have explicitly set to use Ceph during installation.

[Vladimir Kuklin (vkuklin)]

In this case then swift backend should be used.

[Vladimir Kuklin (vkuklin)]

We need some attention from OpenStack developers here.

[Ivan Kolodyazhny (e0ne)]

@Vladimir, what help do you need from Cinder team?

[Bartłomiej Piotrowski (bpiotrowski)]

The root cause is that cinder::backup class doesn't pull its subclasses ::ceph and ::swift to actually configure cinder-backup backend. The patch I'm working on will configure Swift if radosgw hasn't been enabled for block storage; otherwise it will use Ceph and rados.

The main issues I encountered so far were that backuping to Swift takes too long time and not started radosgw on nodes. Even if I start the latter manually, Cinder fails to establish a connection. I'll post further updates tomorrow.

[Bartłomiej Piotrowski (bpiotrowski)]

Cinder fails to connect to RDB due to username/keyring misconfiguration. I've modified ceph puppet module to create a client and a pool for cinder-backup and I'm testing the changes now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179715

[Bartłomiej Piotrowski (bpiotrowski)]

We don't enable cinder-backup in Fuel <6.1.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/179715
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b3fa159e997b2965942a0ad784d07fec3c42619c
Submitter: Jenkins
Branch: master

commit b3fa159e997b2965942a0ad784d07fec3c42619c
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon May 4 09:31:45 2015 +0200

    Configure cinder-backup for Ceph backend

    As the cinder::backup class doesn't configure any backend by default,
    cinder::backup::ceph has to be defined explicitly. Additionally
    cinder-backup requires a seperate client and pool for storing backups.
    The user has to have read access to regular cinder's pool in order to
    actually backup anything.

    Change-Id: I445a03840a7ef17a799d99f161d874df99821eef
    Closes-Bug: 1446593

[Kyrylo Romanenko (kromanenko)]

Verified on
# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "395"
  build_id: "2015-05-08_11-08-49"
  nailgun_sha: "46f55c293e4540d31bcaa6ca3fba77235fb27537"
  python-fuelclient_sha: "af6c9c3799b9ec107bcdc6dbf035cafc034526ce"
  astute_sha: "6a4dcd11c67af2917815f3678fb594c7412a4c97"
  fuel-library_sha: "f385d6a58298c702f8d4f14c452dcffdc0b1e2a3"
  fuel-ostf_sha: "740ded337bb2a8a9b3d505026652512257375c01"
  fuelmain_sha: "3eca5e8f7ca6a83faff5feeca92c21cff31c0af1"

in both envs with

Centos 6.5, CentOS default kernel, Neutron VLAN, QEMU, Ceph RDB for Cinder and Glance, Replication Factor = 2.
3 nodes:
1 Controller + Ceph OSD
1 Compute + Ceph OSD
1 Ceph OSD

and

Ubuntu 14.04.1 , Neutron VLAN, QEMU, Ceph RDB for Cinder and Glance, Replication Factor = 1.
3 nodes:
1 Controller
1 Compute
1 Ceph OSD