Thread leak in cinder-backup with ceph backend

[Mikhail Chernik]

Found on MOS 9.2 snapshot #688

After creating a backup, half of threads created by cinder-backup does not terminate.

This cause huge number of existing threads after a while:

root@node-49:~# ps -T -p $(pgrep cinder-backup) | wc -l
30966

Steps to reproduce:

1. Deploy environment with Ceph RBD as Cinder backend
2. Create an image: cinder create --name test-image 1
3. Note number of threads for cinder-backup: ps -T -p $(pgrep cinder-backup) | wc -l
4. Create a backup for the image: cinder create-backup test-image
5. After backup creation completes note number of threads again

Expected result: number of threads before and after creating backup is the same
Actual result: number of theads after creating backup is bigger (25 more)

The issue is not reproduced on devstack, so it seems to be already fixed in upstream

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: cheng <email address hidden>
Review: https://review.fuel-infra.org/29654

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29654
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: e50d219ae2d63af416dbd03bcf3f88a1d2cdd161
Author: cheng <email address hidden>
Date: Wed Jan 4 16:36:23 2017

Close connection to ceph after cinder bakcup

If we don't close connection to ceph cluster,
The connection count increases with cinder bakcup until exceed max open files.
Then rise OSError: [Errno 24] Too many open files

This patch is to close connection to ceph cluster after cinder backup.

Change-Id: I2fb243d2a57771dc3589e96db54e998e2c1c8ef7
Closes-bug: #1653161
(cherry picked from commit 1d51eb64acebf7bafa277f0060bc5d2c7ca73d52)

[TatyanaGladysheva (tgladysheva)]

Issue is still reproduced on 9.2 snapshot #741.

Actual results:
http://paste.openstack.org/show/594385/

[Dmitry Pyzhov (dpyzhov)]

Fixes are here:
https://review.fuel-infra.org/29737
https://review.fuel-infra.org/#/c/29738/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29738
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 31d512ce4bce552531a7235512f7c55f3d831dc0
Author: Ivan Kolodyazhny <email address hidden>
Date: Thu Jan 12 10:18:24 2017

Re-use RBDImageMetadata and RBDVolumeIOWrapper from os-brick

Closes-Bug: #1653161
Change-Id: Ic359d60939ef15bbdf863f09aef66e4055914417

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #753.

Actual results:
http://paste.openstack.org/show/594804/

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/33412

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Michael Dovgal <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33412
Reason: need to test if the issue https://bugs.launchpad.net/mos/+bug/1653161 is reproducible in this version. Merge it after, if necessary