Ceph OSD is down on one node after deployment, image removing from glance hangs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
New
|
Undecided
|
Oleksiy Molchanov | ||
Mitaka |
New
|
Undecided
|
Oleksiy Molchanov |
Bug Description
Fuel version info (9.0 mos build #458): http://
After successful deployment, OSTF tests which create/remove images (or instance snapshots) fail, because deletion hangs in glance. The cause of that is Ceph OSD failure on one of 5 nodes:
root@node-1:~# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.34991 root default
-2 0.04999 host node-1
0 0.04999 osd.0 down 0 1.00000
-3 0.04999 host node-3
1 0.04999 osd.1 up 1.00000 1.00000
-4 0.04999 host node-2
2 0.04999 osd.2 up 1.00000 1.00000
-5 0.09998 host node-4
3 0.04999 osd.3 up 1.00000 1.00000
6 0.04999 osd.6 up 1.00000 1.00000
-6 0.09998 host node-5
4 0.04999 osd.4 up 1.00000 1.00000
5 0.04999 osd.5 up 1.00000 1.00000
Here is the same issue reported for kilo: https:/
Looks like ceph-osd died or was killed by something during the deployment:
root@node-1:~# service ceph-osd-all status
ceph-osd-all start/running
root@node-1:~# service ceph-all status
ceph-all start/running
root@node-1:~# ps aux | grep [c]eph
root 22524 0.1 1.1 279320 35900 ? Ssl 09:43 0:20 /usr/bin/ceph-mon --cluster=ceph -i node-1 -f
root@node-1:~#
Logs from node-1:
* ceph osd http://
* ceph mon http://
* puppet http://
* upstart-osd http://
* upstart-osd-all http://
After I manually restarted not running OSD, deletion images from glance became to work fine:
I think there are 3 problems here:
1) upstart ceph-* jobs don't detect OSD failures
2) puppet task finished OK despite the fact that OSD wasn't running
3) glance operations (image deletion) just hanged without any errors
Diagnostic snapshot: https:/
Changed in fuel: | |
assignee: | Fuel Sustaining (fuel-sustaining-team) → Oleksiy Molchanov (omolchanov) |
(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:
actual result
expected result
steps to reproduce
For more detailed information on the contents of each of the listed sections see https:/ /wiki.openstack .org/wiki/ Fuel/How_ to_contribute# Here_is_ how_you_ file_a_ bug