Too few ceph pgs on rally tests

[Alexey Yelistratov]

Performed light rally tests from 2015/09/16 14:09:26 to 2015/09/16 16:12:41.

VM instances failed to spawn due to ceph filesystem reading error.

from nova-all.log: http://paste.openstack.org/show/466436/

root@node-3:~# rbd -p images ls -l | grep 6bb86ecf-75ac-47a9-8db9-f72034c9aa96
6bb86ecf-75ac-47a9-8db9-f72034c9aa96 12839k 2
6bb86ecf-75ac-47a9-8db9-f72034c9aa96@snap 12839k 2 yes

rally test log: http://paste.openstack.org/show/466437/

ceph status:

   cluster d9646656-1b8d-4140-89d7-e228a7d72d0f
     health HEALTH_WARN too few pgs per osd (6 < min 20)
     monmap e1: 1 mons at {node-1=192.168.0.8:6789/0}, election epoch 1, quorum 0 node-1
     osdmap e565: 125 osds: 125 up, 125 in
      pgmap v5357: 768 pgs, 12 pools, 495 MB data, 187 objects
            260 GB used, 117 TB / 117 TB avail
                 768 active+clean

Cluster configuration:
baremetal, ubuntu trusty, IBP, HA, neutron-vlan, no DVR, Ceph-all, Nova-debug, Nova-quotas, 7.0-288

api: '1.0'
astute_sha: a717657232721a7fafc67ff5e1c696c9dbeb0b95
auth_required: true
build_id: '288'
build_number: '288'
feature_groups:
- mirantis
fuel-agent_sha: 082a47bf014002e515001be05f99040437281a2d
fuel-library_sha: 121016a09b0e889994118aa3ea42fa67eabb8f25
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 1f08e6e71021179b9881a824d9c999957fcc7045
fuelmain_sha: 6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85
nailgun_sha: 93477f9b42c5a5e0506248659f40bebc9ac23943
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 1ce8ecd8beb640f2f62f73435f4e18d1469979ac
release: '7.0

Diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-16_17-41-10.tar.xz

[Leontii Istomin (listomin)]

the issue also has been reproduced with 47 OSDs and 7.0-296 build
    cluster 8da895d3-f2f7-46e1-a6a3-29ba446df4ae
     health HEALTH_WARN too few pgs per osd (16 < min 20)
     monmap e3: 3 mons at {node-28=192.168.0.3:6789/0,node-34=192.168.0.26:6789/0,node-35=192.168.0.52:6789/0}, election epoch 4, quorum 0,1,2 node-28,node-34,node-35
     osdmap e342: 47 osds: 46 up, 46 in
      pgmap v82746: 768 pgs, 12 pools, 7050 MB data, 1734 objects
            115 GB used, 42452 GB / 42567 GB avail
                 768 active+clean

But hasn't with 3 OSDs and 7.0-288 build
    cluster 39aa7f69-321e-449f-b42f-03b91d497ca2
     health HEALTH_OK
     monmap e5: 5 mons at {node-21=192.168.0.22:6789/0,node-22=192.168.0.16:6789/0,node-23=192.168.0.3:6789/0,node-24=192.168.0.19:6789/0,node-25=192.168.0.20:6789/0}, election epoch 22, quorum 0,1,2,3,4 node-23,node-22,node-24,node-25,node-21
     osdmap e35: 3 osds: 3 up, 3 in
      pgmap v29386: 960 pgs, 15 pools, 17334 MB data, 4619 objects
            34363 MB used, 2757 GB / 2791 GB avail
                 960 active+clean
  client io 93 B/s wr, 0 op/s

[Alexey Yelistratov (ayelistratov-deactivatedaccount)]

Was glance able to read the image from ceph or the same ImageNotFound error occurred?

[Kostiantyn Danylov (kdanylov)]

health HEALTH_WARN is not an error, ceph should works,as usually and serves reads.
Error is in OS.

Incorrect PG number for ceph is known issue

[Mike Fedosin (mfedosin)]

Folks, nova doesn't work with v2 client in glance, but from this log http://paste.openstack.org/show/466436/ I see call to it.

[Denis Meltsaykin (dmeltsaykin)]

As per comment #3 this is a known issue and has no user-impact, I'm closing this bug as Won't fix for 7.0-updates.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug