incorrect pg_num and pgp_num in Ceph

Bug #1294579 reported by Gleb on 2014-03-19
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Dmitry Borodaenko

Bug Description

I have Ceph cluster created by the Fuel 4.0. It has 252 osd but only 100 placement groups for every pool.

root@node-XX:~# ceph osd dump | grep 'rep size'
pool 0 'data' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 8757 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 8760 owner 0
pool 2 'rbd' rep size 3 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 8764 owner 0
pool 3 'images' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 8768 owner 0
pool 4 'volumes' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 8773 owner 0
pool 5 'compute' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 8777 owner 0

root@node-XX:~# ceph -s
  cluster 57491f67-6123-4aef-a8bd-xxxxxxxxxxxx
   health HEALTH_OK
   monmap e3: 3 mons at {node-38=10.135.61.3:6789/0,node-40=10.135.61.5:6789/0,node-41=10.135.61.6:6789/0}, election epoch 24, quorum 0,1,2 node-38,node-40,node-41
   osdmap e8840: 252 osds: 252 up, 252 in
    pgmap v52815: 4488 pgs: 4488 active+clean; 916 GB data, 134 GB used, 228 TB / 228 TB avail
   mdsmap e1: 0/0/1 up

The right number of the placement groups in cluster should be nearly to number of OSD * 100 / replication factor.
(http://ceph.com/docs/master/rados/operations/placement-groups/)

Changed in fuel:
milestone: none → 4.1.1
assignee: nobody → Dmitry Borodaenko (dborodaenko)
importance: Undecided → High
Changed in fuel:
status: New → Confirmed
Andrey Korolyov (xdeller) wrote :

Just pass proper values on the pool creation. Optionally, delete built-in pools with pg_num inherited from osd_pg_bits, they make no sense and just increase amount of calculation during rebalance.

tags: added: backports-4.1.1
Changed in fuel:
milestone: 4.1.1 → 5.0

Fix proposed to branch: master
Review: https://review.openstack.org/82961

Changed in fuel:
status: Confirmed → In Progress
Gleb (gleb-q) on 2014-03-27
tags: added: customer-found

Reviewed: https://review.openstack.org/82961
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=2a3e1288bf1ac6a7576ffc25f666067f307acbb3
Submitter: Jenkins
Branch: master

commit 2a3e1288bf1ac6a7576ffc25f666067f307acbb3
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Mar 25 18:21:26 2014 -0700

    Calculate Ceph pg_num based on number of OSDs

    Flake8 configuration updated for compatibility with flake8 2.1 and
    hacking 0.8.1 (H234 disabled, indentation error fixed in sqlalchemy).

    serializer and serialize() extracted from child classes into
    OrchestratorSerializerTestBase to reduce code duplication.

    Change-Id: I684f7a5e692e8d063435890c61efd3756e254d09
    Partial-bug: #1294579
    Signed-off-by: Dmitry Borodaenko <email address hidden>

Reviewed: https://review.openstack.org/84298
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=99009ba9352c0318c3bc2a5d2b1f9e882b3e418a
Submitter: Jenkins
Branch: stable/4.1

commit 99009ba9352c0318c3bc2a5d2b1f9e882b3e418a
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Mar 25 18:21:26 2014 -0700

    Calculate Ceph pg_num based on number of OSDs

    Flake8 configuration updated for compatibility with flake8 2.1 and
    hacking 0.8.1 (H234 disabled, indentation error fixed in sqlalchemy).

    serializer and serialize() extracted from child classes into
    OrchestratorSerializerTestBase to reduce code duplication.

    Change-Id: I684f7a5e692e8d063435890c61efd3756e254d09
    Partial-bug: #1294579
    Signed-off-by: Dmitry Borodaenko <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
milestone: 5.0 → 4.1.1
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/82965
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2e604f4bef5b7f7ba96d2f3680dfab8d2e76bfd1
Submitter: Jenkins
Branch: master

commit 2e604f4bef5b7f7ba96d2f3680dfab8d2e76bfd1
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Mar 25 19:14:29 2014 -0700

    Set Ceph pg_num dynamically based on fuel_settings

    Depends on the following fuel-web commit:
    https://review.openstack.org/82961

    Change-Id: I7095d2600b279c1225c926e61183ed0e81180a73
    Closes-bug: #1294579
    Signed-off-by: Dmitry Borodaenko <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/85111
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8cc55690c93633cefd0f3ba6493c8d949fbcbb51
Submitter: Jenkins
Branch: stable/4.1

commit 8cc55690c93633cefd0f3ba6493c8d949fbcbb51
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Mar 25 19:14:29 2014 -0700

    Set Ceph pg_num dynamically based on fuel_settings

    Depends on the following fuel-web commit:
    https://review.openstack.org/82961

    Change-Id: I7095d2600b279c1225c926e61183ed0e81180a73
    Closes-bug: #1294579
    Signed-off-by: Dmitry Borodaenko <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Anastasia Palkina (apalkina) wrote :

Verified on ISO #344
"build_id": "2014-06-05_15-25-50",
"mirantis": "yes",
"build_number": "344",
"nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b",
"ostf_sha": "2b7b39e4b6ea89751b65171f24a8e80b5cac56aa",
"fuelmain_sha": "9964da7dec34d3100419c1c77c8f5235d8e30f14",
"astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
"release": "4.1B",
"fuellib_sha": "3511461a2b529619a787a6306441d9039699e71d"

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers