Change pg_num pools parameter when adding osd nodes

[Andrey Grebennikov]

When deploying MOS with Ceph cluster, pg_num and pgs_num parameters to be only calculated once (when creating the pools). We need to recalculate it every time we add new osd nodes, as well as we have to execute "ceph osd pool set {pool-name} pg_num {pg_num}" and "ceph osd pool set {pool-name} pgp_num {pgp_num}" for all existing pools.

[Dmitry Borodaenko (angdraug)]

Lets document this in Ops Guide for 6.1, then create/find a BP to implement this properly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179253

[Alex Schultz (alex-schultz)]

Created blueprint https://blueprints.launchpad.net/fuel/+spec/ceph-add-node-update-placement-groups for this.

[Andrey Grebennikov (agrebennikov)]

What I got from conversation with Andrey Korolyov was that we Don't have to change it Every time.
Here is the formula:
pg_num = osd_num * 100.0 / repl
but there is another requirement - "pg_num ~ 2^N", which means we only need to change pg_num when it has to be doubled.
If we'll be changing pg_num every time it will cause cluster unavailability for some time because it will completely restructurize data.
I guess we have to look closer one more time on this bug and proposed solution.

[Alex Schultz (alex-schultz)]

My understanding from researching this is that it's not something that necessarily needs to be done all the time. It looks like it would only be something you would do if you are significantly changing the size of the ceph cluster and realistically have >50 OSDs. At the moment I'm documenting how to do it for the operations guide, and I've kicked off a blueprint to start looking into this further if we want to add it into the deployment process.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/179253
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=1d2892c239d8e20b361c0b7c9ca7bfb9eb81ddc1
Submitter: Jenkins
Branch: master

commit 1d2892c239d8e20b361c0b7c9ca7bfb9eb81ddc1
Author: Alex Schultz <email address hidden>
Date: Thu Apr 30 16:25:18 2015 -0500

    How To: Adjust Placement Groups for new Ceph Nodes

    Currently fuel only calculates pg_num and pgp_num for ceph pools as part
    of the initial deployment. When additional nodes are added to an
    environment it may be necessary to update these values for the ceph
    deployment for optimal performance. This document explains how to do
    this manually until this functionality can be rolled into the fuel
    deployment process.

    Change-Id: I7ed502a1665ea314ab3dd4f868eca865fe550961
    Partial-Bug: 1383931

[Dmitry Borodaenko (angdraug)]

Now that the documentation workaround is merged, the remaining work should be tracked in the associated blueprint https://blueprints.launchpad.net/fuel/+spec/ceph-add-node-update-placement-groups -- the bug is downgraded to Medium and postponed to 7.0.

[Roman Podoliaka (rpodolyaka)]

We no longer fix Medium bugs in 8.0, closing as Won't Fix

[Alexei Sheplyakov (asheplyakov)]

> We need to recalculate it every time we add new osd nodes

This statement is not quite true. Splitting placement groups causes a massive data rebalancing and can improve
the performance and data safety only if the number of OSDs has radically changed (say, >= x4).
Splitting PGs after adding each OSD is definitely a bad idea. Marking bug as Invalid