[Ceph] Fuel must update (all) ceph.conf [mons] on ceph monitor re-deploy

[Miroslav Anashkin]

Usual scenario:

HA + Ceph MONs either on controllers or on separate nodes

One of controllers or Ceph MON nodes has failed and need re-deployment.

Customer deploys new controller.

However - /etc/ceph/ceph.conf is created in scope of primary controller/Ceph MON deployment only and remains untouched.

As a result - Ceph continue to complain about the missing Ceph monitor, since it remains in ceph.conf
New Ceph monitor is not deployed at all - since it does not exist in ceph.conf

[Bogdan Dobrelya (bogdando)]

Please provide release version

[Miroslav Anashkin (manashkin)]

We encountered this issue with 6.0 at the customer side.
However - 6.1 and 7.0 still lacks this functionality as well.

[Vitaly Sedelnik (vsedelnik)]

Miroslav, let's wait for feedback from Ceph and Fuel teams but that looks like as feature request not a bug. I am not sure we should implement this in already released versions.

[Vitaly Sedelnik (vsedelnik)]

We don't support master node updates in 5.1/5.1.1 and 6.0 maintenance updates. Setting to Invalid for 5.1.1 and 6.0 updates.

[Andrew Woodward (xarses)]

We need more information about what will occur if the monitors list is partially out of date (some are correct, some are not).

Is it warnings? performance deg?

Current description doesn't justify High

[Timur Nurlygayanov (tnurlygayanov)]

Marked as Confirmed for MOS 6.1 and MOS 7.0

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 5.1.1-updates and 6.0-updates because Fuel fixes cannot be delivered in maint updates for these version

[Julia Portnova (yportnova)]

Won't fix for 7.0

This fix requires a lot of changes as for each adding/deleting mon we need to update ceph.conf and restart all ceph-osds and mons on each nodes.

[Kostiantyn Danylov (kdanylov)]

Before restart osd need to set noout flag

# ceph osd set noout

After osd restarted this flag need to be removed

# ceph osd unset noout

[Matthew Mosesohn (raytrac3r)]

We can't fix ceph osd + controller combined very easily because that has a warning already:
"Ceph data still exists on: node-12. You must manually remove the OSDs from the cluster and allow Ceph to rebalance before deleting these nodes.
Deployment has failed. Method granular_deploy. Deployment failed on nodes 13.
Inspect Astute logs for the details"

I'll try to address the controller (with ceph mon) and separate OSD scenario

[Matthew Mosesohn (raytrac3r)]

Ceph conf on controllers (which run ceph-mon) remove deleted nodes from mon_host config correctly when we redeploy other controllers. There are no recurring messages about any missing nodes on removal of primary controller or deployment of additional controller. This bug is invalid for 8.0.