Backup w/ ceph hangs when creating a backup

[Nate Potter]

On a fresh install of kolla using cinder and ceph as a backend, the backup service isn't able to create a backup of a volume. The service itself starts up fine, but when you ask for a backup to be created, it notes that the creation starts but the privsep daemon doesn't start up and the work of backing up isn't actually done.

Here is a copy of my backup logs using kolla: http://paste.openstack.org/show/558606/

Ans here is a copy of backup logs from a working deployment: http://paste.openstack.org/show/558537/

You can see that in both cases it says it's running the privsep helper, but for my kolla deployment the actual startup doesn't happen.

[Eduardo Gonzalez (egonzalez90)]

Hi, which Kolla version and OS you used in both deployments? Can you paste cinder-api and nova-compute logs?

[Nate Potter (ntpttr)]

Hi Eduardo, I'm using Kolla master and Ubuntu 14.04 for my deployment. Here is the cinder-api log, looks like it just got back a 202. There didn't seem to be anything in the nova compute logs except for periodic polling http://paste.openstack.org/show/560834/

[Nate Potter (ntpttr)]

I thought it might have been that oslo.privsep isn't being installed like many other oslo services in openstack-base, becasue if I tried to run 'privsep-helper' in the containers I would get command not found. I added it in, and the command showed up in the containers, but the daemon still didn't start up so it didn't seem to solve the issue.

[Steven Dake (sdake)]

may be a problem with the sudoers file. Would you create a log of the commands you run to generate this problem?

run the command script
run your workload that generates this problem
run the command exit

this will create a file called typescript which I'd ask you to attach to the bug tracker.

This report is a little vague on details; this command log will help the developers replicate and fix the problem.

[Jeffrey Zhang (jeffrey4l)]

could u try this again, it should be already fixed.

[franklin798 (franklin798)]

I meet the same issue.My env is kolla stable/ocata But the log show that,the Privsep daemon failed to start.
here is log for cinder-backup
Create backup started, backup: ecd96f90-1020-42a0-8336-cd1ea9c1062c volume: 59483280-214f-4d32-9788-ca93c3d7d172.
2017-03-24 18:10:34.563 7 DEBUG os_brick.utils [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '172.16.0.211', 'enforce_multipath': False, 'host': None, 'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'multipath': False}" trace_logging_wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/os_brick/utils.py:146
2017-03-24 18:10:34.565 7 INFO oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp_bhDd5/privsep.sock']
2017-03-24 18:10:35.849 7 WARNING oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] privsep log: [Errno 1] Operation not permitted
2017-03-24 18:10:35.901 7 INFO oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] Spawned new privsep daemon via rootwrap
2017-03-24 18:10:35.902 7 DEBUG oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] Accepted privsep connection to /tmp/tmp_bhDd5/privsep.sock __init__ /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py:313
2017-03-24 18:10:35.903 7 INFO oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] privsep daemon starting
2017-03-24 18:10:35.903 7 INFO oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] privsep process running with uid/gid: 0/0
2017-03-24 18:10:35.904 7 ERROR oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] [Errno 1] Operation not permitted
2017-03-24 18:10:35.904 7 DEBUG oslo_privsep.comm [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] EOF on privsep read channel _reader_main /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/comm.py:141
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon [req-d85ff49b-10e8-4ea6-a949-3eed60aa082a - - - - -] Error while sending initial PING to privsep: [Errno 32] Broken pipe
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon Traceback (most recent call last):
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 165, in exchange_ping
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon reply = self.send_recv((Message.PING.value,))
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/comm.py", line 160, in send_recv
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon self.writer.send((myid, msg))
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/comm.py", line 55, in send
2017-03-24 18:10:35.905 7 ERROR oslo.privsep.daemon self.writesock.sendall(buf)
2017-03-24 18:10:35.905 ...

[Jaesang Lee (jaesang-lee)]

This is capabilities issue in cinder-backu pod. If cinder-backup container doesn't have SYS_ADMIN capability, privsep daemon can't start.
Look at this patch, https://review.openstack.org/#/c/506033/

[Jeffrey Zhang (jeffrey4l)]

In kolla, we run kolla-backup with privileged: True, so this should work.

[Ravinder Kumar (rhcayadav)]

in queens using latest ubuntu_binary images ( i have checked taking backup , its working fine)

(keystone)[root@orioncn3 /]# openstack volume create --size 8 yadav
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2018-06-12T01:36:39.000000 |
| description | None |
| encrypted | False |
| id | 5ff9b00f-9a0c-4506-ab4c-2b64f05a377f |
| migration_status | None |
| multiattach | False |
| name | yadav |
| properties | |
| replication_status | None |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | None |
| updated_at | None |
| user_id | 2611ce4936804a758dddf2dc29a5a799 |
+---------------------+--------------------------------------+

(keystone)[root@orioncn3 /]# openstack volume backup create yadav
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | 060e303f-b5a4-429a-b862-836d3e13f30b |
| name | None |
+-------+--------------------------------------+

(keystone)[root@orioncn3 /]# openstack volume backup list
+--------------------------------------+------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+-----------+------+
| 060e303f-b5a4-429a-b862-836d3e13f30b | None | None | available | 8 |
+--------------------------------------+------+-------------+-----------+------+
(keystone)[root@orioncn3 /]#