[library] ERR in puppet.log: RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite

[Anastasia Palkina]

"build_id": "2014-06-11_00-30-36",
"mirantis": "yes",
"build_number": "250",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "f851199db57d9561b51c429956c056b463ad3f0b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "e59cb325d99fba49085ed5b0b6cfb0d9a83d2fc8",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "d9b8daa61e131e3c346e93d3870620897a71c593"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Choose both ceph
4. Choose rados
5. Add 3 controller+ceph, 2 compute+ceph
6. Start deployment. It was successful
7. There are error in puppet.log on controllers (node-1,2,3):

Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] Traceback (most recent call last):
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/util/decorators.py", line 10, in inner
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] def inner(*args, **kwargs):
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/conf.py", line 12, in write_conf
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] line = self.fp.readline()
Wed Jun 11 10:52:16 +0000 2014 /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns (notice): [node-1][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite

[Dmitry Borodaenko (angdraug)]

This error message is generated by ceph-deploy and is safely ignored by puppet. It does not affect deployment, and is likely to go away with new versions of ceph-deploy.

[Ryan Moe (rmoe)]

In ceph-deploy 1.5.9 this error will cause OSD prepare to fail. ceph-deploy 1.2.7 used a context manger to decorate remote methods before calling them. The decorator would catch exceptions from the remote method and raise a new exception with a specific attribute set. The context manager would check for that specific attribute and then log (but not re-raise) the exceptions from the remote methods (see: https://github.com/ceph/ceph-deploy/blob/v1.2.7/ceph_deploy/util/context.py#L58-L63).

[Dmitry Borodaenko (angdraug)]

The problem would only happen if radosgw is enabled (radosgw.pp is the only manifest that modifies ceph.conf outside of primary mon), AND if ceph-osd role is assigned to secondary controllers. Here's what happens:

1) ceph-deploy config pull downloads ceph.conf from primary mon
2) all other ceph-deploy invokations done for secondary controller role are not affected since ceph.conf is not touched until:
3) radosgw.pp (running as part of controller role) modifies ceph.conf to update radosgw\host variable
4) there's no more ceph-deploy invokations done after radosgw.pp for controller, so controller deployment succeeds
5) ceph-osd role starts with ceph-deploy osd prepare which fails due to ceph.conf in /root and /etc/ceph being out of sync after radosgw.pp updates in (3)

The fix is to make sure /root/ceph.conf is updated after /etc/ceph/ceph.conf is modified before any other ceph-deploy commands.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/110168

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110168
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c2401c2b8f28129b5e063e6c8050d3868ba5c02d
Submitter: Jenkins
Branch: master

commit c2401c2b8f28129b5e063e6c8050d3868ba5c02d
Author: Dmitry Borodaenko <email address hidden>
Date: Mon Jul 28 16:39:25 2014 -0700

    create /root/ceph.conf symlink on all ceph nodes

    We need /root/ceph.conf to link to /etc/ceph/ceph.conf on all ceph nodes, not
    just primary monitor. With ceph-deploy 1.3+, if Puppet modifies
    /etc/ceph/ceph.conf and makes it differ from /root/ceph.conf (e.g. change host
    for radosgw), all subsequent ceph-deploy commands will fail.

    Change-Id: I6254c35f546cd22837d29e952b8632759d5b40f8
    Closes-Bug: #1328881

[Dmitry Borodaenko (angdraug)]

ceph-deploy in 5.0.x is 1.2.x, so this bug does not exist there.