Cinder-backup fails when cinder version isn't caught up to the rest of the deployment

[Nate Potter]

During testing of rolling upgrades, one of the scenarios I had was the cinder service starting at stable/mitaka and the rest of the OpenStack services running at master, with the cinder-backup service running using the swift driver. In this setup, trying to create a backup of a volume fails and produces a backup in error status.

Here is the error in the c-bak log:

http://paste.openstack.org/show/565275/

[Michal Dulko (michal-dulko-f)]

Interesting! Are c-bak and c-vol on a single node? Can you try to reproduce that while having both Mitaka's and Newton's volume.filters files in /etc/cinder/rootwrap.d? Simply put both in that directory as for example volume-mitaka.filters and volume-newton.filters - oslo.rootwrap will load both.

[Nate Potter (ntpttr)]

Hi Michal, yes c-bak and c-vol were on the same node in this case - I'll run those tests and get back to you here

[Nate Potter (ntpttr)]

Okay, I added both files to /etc/cinder/rootwrap.d and restarted all of the cinder services, but the problem persists

[Michal Dulko (michal-dulko-f)]

Oh, this looks like os.brick problem. Between Mitaka and Newton we've switched os.brick to use oslo.privsep. Maybe this is related to that fact?

[Nate Potter (ntpttr)]

Yes I'd say that's likely causing the issue, no call was ever made to swift and there's nothing in the swift logs, so it's not happening at a driver level for sure. Looks just related to privsep.

[Sean McGinnis (sean-mcginnis)]

Not sure what we can do here.

The issue appears to be that the installation of Newton upgrades os-brick to the newer version that uses privsep. Patch https://review.openstack.org/#/c/331973/ made the changes necessary for newton and later versions to initialize privsep appropriately, but that does not exist and can't be easily backported to the mitaka code.

The workaround is to get the backup service upgraded before attempting to backup when it's on the same node as the other Cinder services. Or run in an isolated venv perhaps.

Might just need to document this upgrade constraint and move on.

[Michal Dulko (michal-dulko-f)]

@Sean: I agree. I'll be writing some Cinder upgrade guide in early Ocata and I'll make sure to emphasize that deploying each service in a separate container or at least using venvs to avoid dependency hell is recommended.