Cinder-backup fails when cinder version isn't caught up to the rest of the deployment

Bug #1619004 reported by Nate Potter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Won't Fix
Medium
Unassigned
os-brick
Invalid
Medium
Unassigned

Bug Description

During testing of rolling upgrades, one of the scenarios I had was the cinder service starting at stable/mitaka and the rest of the OpenStack services running at master, with the cinder-backup service running using the swift driver. In this setup, trying to create a backup of a volume fails and produces a backup in error status.

Here is the error in the c-bak log:

http://paste.openstack.org/show/565275/

Revision history for this message
Michal Dulko (michal-dulko-f) wrote :

Interesting! Are c-bak and c-vol on a single node? Can you try to reproduce that while having both Mitaka's and Newton's volume.filters files in /etc/cinder/rootwrap.d? Simply put both in that directory as for example volume-mitaka.filters and volume-newton.filters - oslo.rootwrap will load both.

Revision history for this message
Nate Potter (ntpttr) wrote :

Hi Michal, yes c-bak and c-vol were on the same node in this case - I'll run those tests and get back to you here

Revision history for this message
Nate Potter (ntpttr) wrote :

Okay, I added both files to /etc/cinder/rootwrap.d and restarted all of the cinder services, but the problem persists

Revision history for this message
Michal Dulko (michal-dulko-f) wrote :

Oh, this looks like os.brick problem. Between Mitaka and Newton we've switched os.brick to use oslo.privsep. Maybe this is related to that fact?

Revision history for this message
Nate Potter (ntpttr) wrote :

Yes I'd say that's likely causing the issue, no call was ever made to swift and there's nothing in the swift logs, so it's not happening at a driver level for sure. Looks just related to privsep.

Changed in cinder:
importance: Undecided → Medium
Changed in os-brick:
importance: Undecided → Medium
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Not sure what we can do here.

The issue appears to be that the installation of Newton upgrades os-brick to the newer version that uses privsep. Patch https://review.openstack.org/#/c/331973/ made the changes necessary for newton and later versions to initialize privsep appropriately, but that does not exist and can't be easily backported to the mitaka code.

The workaround is to get the backup service upgraded before attempting to backup when it's on the same node as the other Cinder services. Or run in an isolated venv perhaps.

Might just need to document this upgrade constraint and move on.

Changed in os-brick:
status: New → Invalid
Revision history for this message
Michal Dulko (michal-dulko-f) wrote :

@Sean: I agree. I'll be writing some Cinder upgrade guide in early Ocata and I'll make sure to emphasize that deploying each service in a separate container or at least using venvs to avoid dependency hell is recommended.

Changed in cinder:
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.