[library] Command exceeded timeout at /etc/puppet/modules/ceph/manifests/osd.pp:33

[Anastasia Palkina]

"build_id": "2014-04-07_04-17-26",
"mirantis": "yes",
"build_number": "270",
"nailgun_sha": "7a05e365240ab27c492b20585ef8ac8557102cc0",
"ostf_sha": "de0222fed646525d248dc6892eeceab139d5c469",
"fuelmain_sha": "5714f2a3cf56ebdc124e9555dfff0fe274e84bdd",
"astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
"release": "4.1",
"fuellib_sha": "52e7f57695f33bafa5d84d524d77f1bc3a2289b2"

1. Create new environment (CentOS, simple mode)
2. Choose GRE neutron
3. Choose both Ceph and installing ceilometer
4. Add controller, compute and 3 ceph
5. Start deployment. It was successful
6. But there is error in puppet logs on first ceph node:

(/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) change from notrun to 0 failed: Command exceeded timeout at /etc/puppet/modules/ceph/manifests/osd.pp:33

But ceph is alive
[root@node-15 ~]# ceph osd tree
# id weight type name up/down reweight
-1 0.36 root default
-2 0.12 host node-18
0 0.06 osd.0 up 1
4 0.06 osd.4 up 1
-3 0.12 host node-17
1 0.06 osd.1 up 1
3 0.06 osd.3 up 1
-4 0.12 host node-19
2 0.06 osd.2 up 1
5 0.06 osd.5 up 1

[root@node-15 ~]# ceph -s
  cluster bc868fd3-d7ac-4e9c-ab62-c3f162925192
   health HEALTH_OK
   monmap e1: 1 mons at {node-15=192.168.0.6:6789/0}, election epoch 2, quorum 0 node-15
   osdmap e58: 6 osds: 6 up, 6 in
    pgmap v114: 492 pgs: 492 active+clean; 14464 KB data, 12549 MB used, 368 GB / 380 GB avail
   mdsmap e1: 0/0/1 up

[Andrew Woodward (xarses)]

This happens on occasion and is likely the result of the older version of ceph-deploy. We should update the version of ceph-deploy or add retries = number of osd's on this system. Puppet will re-run and pass on the next iteration.

[Anastasia Palkina (apalkina)]

Reproduced for version 5.0.1

"build_id": "2014-07-08_13-57-45",
"mirantis": "yes",
"build_number": "107",
"ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
"nailgun_sha": "c0082e3a0e8544bad7bd45c15c5dd8632ea045b5",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "b0f5151d12751b9b55dcd69bd1445d0d480012d6",
"astute_sha": "a4edb51661f50c66e247e0b8d00f2d01e0658fe6",
"release": "5.0.1",
"fuellib_sha": "d4cb36208efaf51a7c0ca012fa63d596d4ee2e29"

1. Create new environment (CentOS, simple mode)
2. Choose both Ceph
3. Choose Ceilometer
4. Add controller, compute+cinder+ceph, cinder+mongo, ceph
5. Start deployment. It was successful
6. But there is error in puppet.log on node with single ceph (node-12):

2014-07-08 14:00:31 ERR

 (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) change from notrun to 0 failed: Command exceeded timeout

Ceph is alive:

[root@node-9 ~]# ceph osd tree
# id weight type name up/down reweight
-1 0.24 root default
-2 0.12 host node-10
0 0.06 osd.0 up 1
1 0.06 osd.1 up 1
-3 0.12 host node-12
2 0.06 osd.2 up 1
3 0.06 osd.3 up 1

[root@node-9 ~]# ceph -s
  cluster dd4ceee6-0ce0-4ae1-86b0-d6ae9ad10d99
   health HEALTH_OK
   monmap e1: 1 mons at {node-9=192.168.0.2:6789/0}, election epoch 1, quorum 0 node-9
   osdmap e35: 4 osds: 4 up, 4 in
    pgmap v71: 960 pgs: 960 active+clean; 14464 KB data, 8367 MB used, 245 GB / 253 GB avail; 112B/s rd, 0op/s
   mdsmap e1: 0/0/1 up

[Dmitry Borodaenko (angdraug)]

We now have a more recent ceph-deploy 1.5.9 in 5.1 so this problem should disappear. Since it's a low priority bug it will not be backported to stable release series.

[Anastasia Palkina (apalkina)]

Verified on ISO #436
"build_id": "2014-08-14_02-01-17",
"ostf_sha": "d2a894d228c1f3c22595a77f04b1e00d09d8e463",
"build_number": "436",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "b5bdd19c2dbeb26ce3bd88270d09f5e7541a3aea",
"production": "docker",
"fuelmain_sha": "9f327045cdd72d406d89063393a499635be5e3d4",
"astute_sha": "b52910642d6de941444901b0f20e95ebbcb2b2e9",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "4b085bfbf7be973f0aa29d9d5e4f3ebd5bf789a1"