Upgrading ceph-osd from Kilo to Mitaka "misses" /var/lib/ceph ownership change

[Rafael David Tinoco]

I just upgraded ceph-{osd,mon} from Kilo to Mitaka (when doing a full OpenStack upgrade)
and it looks like the charm missed the step described in:

http://docs.ceph.com/docs/master/release-notes/#upgrading-from-hammer

Fix the ownership:

chown -R ceph:ceph /var/lib/ceph
chown -R ceph:ceph /var/log/ceph

As you can see in the log files:

http://paste.ubuntu.com/22719924/
http://paste.ubuntu.com/22719976/

Fixing the ownership by hand makes the error:

 No block devices detected using current configuration

To disappear from juju status.

[Rafael David Tinoco (rafaeldtinoco)]

I just updated the charm by trigger the upgrade process changing the source:

juju set ceph-mon source='trusty-proposed/mitaka'
juju set ceph-osd source='trusty-proposed/mitaka'

From the charm.

[Edward Hope-Morley (hopem)]

I think that the package itself should be responsible for ensuring that the path is accessible by the version ceph it is installing. The charm can/should certainly implement a workaround as well though.

[Chris Holcombe (xfactor973)]

I actually committed a fix for this. I hit this also myself. Here's the patch: https://review.openstack.org/#/c/345646/ for ceph-mon and https://review.openstack.org/#/c/345647/ for ceph-osd. Just be aware that if you have a lot of data in your cluster this could be extremely slow or fail.

[Chris Holcombe (xfactor973)]

I agree with you Ed. The package should ensure this. The charm fix is a band-aid.

[Rafael David Tinoco (rafaeldtinoco)]

+1

[James Page (james-page)]

Actually the package *can't* do this - the packaging is configured not to restart or attempt to manage the ceph daemons during normal package updates; the reason being is that you need to co-ordinate this just like any other outage, and doing it during a package upgrade is not ideal and can cause huge lag in a large cluster during the package update transaction.

[James Page (james-page)]

Rafael

Which charm versions where you using? the ownership during upgrade change only just landed in the stable charms (as of end of July) so is it possible that you had an older charm version at the point of ceph upgrade?

[Rafael David Tinoco (rafaeldtinoco)]

Hello James,

I'm creating and destroying this deployment every day (trying to reproduce other bug). For this particular error, the environment was created a few hours before using:

$ cat ceph-osd.yaml
ceph-osd:
    source: 'cloud:trusty-proposed/kilo'
    osd-devices: /dev/vdb
    osd-reformat: "yes"

juju deploy --config=$CONFIGDIR/ceph-osd.yaml --to $tkcompute01 cs:trusty/ceph-osd ;
juju add-unit --to $tkcompute02 ceph-osd ; sleep 5
cmd juju add-unit --to $tkcompute03 ceph-osd ; sleep 5

[Rafael David Tinoco (rafaeldtinoco)]

I'm re-deploying everything again.. upgrading nova-compute right now. Will update ceph again in a few more minutes. Let me know if you need anything from my setup.

[Rafael David Tinoco (rafaeldtinoco)]

Ok I just re-deployed everything.

After I finished upgrade, I rebooted all the environment.

After ceph machines came back, I had:

https://pastebin.canonical.com/162720/

So I guess something is wrong... 2nd time I was able to reproduce the same behaviour.

When changing permissions:

juju ssh ceph-osd/0 -- sudo chown -R ceph:ceph /var/lib/ceph
juju ssh ceph-osd/1 -- sudo chown -R ceph:ceph /var/lib/ceph
juju ssh ceph-osd/2 -- sudo chown -R ceph:ceph /var/lib/ceph

juju ssh ceph-osd/0 -- sudo chown -R ceph:ceph /var/log/ceph
juju ssh ceph-osd/1 -- sudo chown -R ceph:ceph /var/log/ceph
juju ssh ceph-osd/2 -- sudo chown -R ceph:ceph /var/log/ceph

Everything went back to normal:

https://pastebin.canonical.com/162725/

[James Page (james-page)]

Interestingly the packaging should be trying to make some directory changes:

       # 5. adjust file and directory permissions
       if ! dpkg-statoverride --list $SERVER_HOME >/dev/null
       then
           chown $SERVER_USER:$SERVER_GROUP $SERVER_HOME
           chmod u=rwx,g=rx,o= $SERVER_HOME
       fi
       if ! dpkg-statoverride --list /var/log/ceph >/dev/null
       then
           chown -R $SERVER_USER:$SERVER_GROUP /var/log/ceph
           # members of group ceph can log here, but cannot remove
           # others' files. non-members cannot read any logs.
           chmod u=rwx,g=rwxs,o=t /var/log/ceph
       fi

       # 6. fix /var/run/ceph
       if [ -d /var/run/ceph ]; then
           chown $SERVER_USER:$SERVER_GROUP /var/run/ceph
       fi

[James Page (james-page)]

Hi Rafael

I tried reproducing your issue this morning - found a different problem (bug 1611719) but that's unrelated to this - I did end up with a ceph deployment that had the correct permissions.

I deployed trusty-kilo; upgraded to trusty-liberty and then to trusty-mitaka.

[James Page (james-page)]

I see you appear to go direct from kilo->mitaka

I'll try that and see if that makes a difference.

[James Page (james-page)]

ceph-mon log for direct upgrade attempt:

2016-08-10 11:10:27 INFO juju-log old_version: cloud:trusty-kilo
2016-08-10 11:10:27 INFO juju-log new_version: cloud:trusty-mitaka
2016-08-10 11:10:27 INFO juju-log Invalid upgrade path from cloud:trusty-kilo to cloud:trusty-mitaka. Valid paths are: ['cloud:trusty-liberty -> cloud:trusty-mitaka', 'cloud:trusty-juno -> cloud:trusty-kilo', 'cloud:trusty-kilo -> cloud:trusty-liberty']

[James Page (james-page)]

ditto on ceph-osd:

2016-08-10 11:13:17 INFO juju-log old_version: cloud:trusty-kilo
2016-08-10 11:13:17 INFO juju-log new_version: cloud:trusty-mitaka
2016-08-10 11:13:18 INFO juju-log Invalid upgrade path from cloud:trusty-kilo to cloud:trusty-mitaka. Valid paths are: ['cloud:trusty-liberty -> cloud:trusty-mitaka', 'cloud:trusty-juno -> cloud:trusty-kilo', 'cloud:trusty-kilo -> cloud:trusty-liberty']

[James Page (james-page)]

Hmm I wonder whether this is due to the co-location of ceph-osd with nova-compute; which order did you perform the upgrade in? switching openstack-origin on the nova-compute node will trigger a package upgrade for all packages on the machine, including ceph.

ceph-osd can be more selective, so the upgrade process will only upgrade ceph packages during the rolling upgrade process.

So normal order would have to be ceph-osd first, and then nova-compute.

Also, looking at the code, the ceph-mon and ceph-osd charms will not support charm driven upgrades from proposed pockets - the map of supported paths is as follows:

# A dict of valid ceph upgrade paths. Mapping is old -> new
upgrade_paths = {
    'cloud:trusty-juno': 'cloud:trusty-kilo',
    'cloud:trusty-kilo': 'cloud:trusty-liberty',
    'cloud:trusty-liberty': 'cloud:trusty-mitaka',
}

So setting anything other than values in that map will not trigger any upgrade code in the charm.

[James Page (james-page)]

(just a hunch but I suspect your ceph-mon units never actually upgraded).

[James Page (james-page)]

Marking Ubuntu task as invalid - this problem its not in the packaging.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/417616

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417616
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=a60775be188a6450f1ec3c8486aad92032acf1d0
Submitter: Jenkins
Branch: master

commit a60775be188a6450f1ec3c8486aad92032acf1d0
Author: James Page <email address hidden>
Date: Sat Jan 7 16:07:36 2017 +0000

    Generalize upgrade paths for osds

    Make use of new charms.ceph utils to generalize the upgrade
    paths for OSD upgrades, ensuring that only supported upgrade
    paths are undertaken for Ubuntu 16.04 UCA pockets.

    Partial-Bug: 1611082

    Change-Id: Ifbf3a7ffbb5ab17e839099658c7a474784ab4083