Ceph monitors not removed when deleting controller

[Jon Skarpeteig]

ceph --admin-daemon /var/run/ceph/ceph-mon.node-16.asok mon_status

Reports:

"mons": [
            { "rank": 0,
              "name": "node-10",
              "addr": "10.0.0.4:6789\/0"},
            { "rank": 1,
              "name": "node-15",
              "addr": "10.0.0.9:6789\/0"},
            { "rank": 2,
              "name": "node-16",
              "addr": "10.0.0.10:6789\/0"}]}}

Although node-10 and node-15 has been deleted through Fuel UI.

This in turn makes rados commands (Puppet) time out on deployment of replacement controller nodes.

[Stanislav Makar (smakar)]

Thanks for bug
Please provide Fuel version and it would be fine if you can upload diagnostic snapshot?

[Jon Skarpeteig (jskarpet)]

Fuel version 6.0

Snapshot: https://cisco.box.com/s/vh291qg3jshsfuayq032jmlzeu1l5fc9

[Vladimir Kuklin (vkuklin)]

Stas, I think we can work on this issue without diagnostic snapshot.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172457

[Stanislav Makar (smakar)]

above patch is a temporary solution due to we do not have granular way for node
removal. As soon as we have it we will do it as separate task.

What is left - update the config file (/etc/ceph/ceph.conf) on left nodes

[Stanislav Makar (smakar)]

but if we deploy new node after deletion, ceph.conf will be fixed

[Stanislav Makar (smakar)]

Custom iso is building http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_6.1_iso/1169/
Would be fine to test before merge

[Vladimir Sharshov (vsharshov)]

Main test scenario:

- deploy HA-cluster (ubuntu or centos) with 3 controllers and 2 ceph nodes (Glance, Cinder);
- after success deployment mark one of controller for deletion;
- run deletion.

Expecting result:

- node remove success;
- in Astute log you can see this messages:
- - Removing ceph mons ["node-<X>"] from cluster
- - Ceph mons are left in cluster: ["node-<X>, ..., node-<X>"]
- - shell command `ceph -f json mon dump` on the one of remaining controllers returns info only about remaining controllers, e.g.:
{"epoch":5,"fsid":"c7fc3eaf-ead2-4bd4-ae3a-61f5db6a236c","modified":"2015-05-07 11:03:06.153854","created":"0.000000","mons":[{"rank":0,"name":"node-6","addr":"192.168.0.11:6789\/0"}],"quorum":[0]}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172457
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=80d33dd44ec09340c275ea5ee7200d1ef47dd467
Submitter: Jenkins
Branch: master

commit 80d33dd44ec09340c275ea5ee7200d1ef47dd467
Author: Stanislav Makar <email address hidden>
Date: Fri Apr 10 13:56:11 2015 +0000

    Fix the problem with removing ceph-mon from cluster

    It is a temporary solution due to we do not have granular way for node
    removal. As soon as we have it we will do it as separate task.

    Change-Id: Ib77227c94a3634f7c91a4d7af4c1ff11ef525553
    Closes-bug: #1435824

[Dmytro Iurchenko (diurchenko)]

Verified on Ubuntu. Monitors are removed and added correctly according to corresponding operations on the contollers.