[doc] Ceph OSD disks are lost at node reboot

[Dmitriy Novakovskiy]

In one of customer deployments we faced the situation when host OS kernel would initialize multiple backplanes with disks in random order at boot time. This causes OSD disks that Fuel deployed w/ ceph-deploy utility to be lost from OSD after node reboot, since they were mounted via /dev/sdXXX pointers and these numbers become different at every boot.

The solution is to mount OSD disks using UUID instead of sdXXX names. It's needs to be checked whether ceph-deploy utility can do it (so Fuel could just pass additional parameter) or a more sophisticated approach is needed to solve this problem.

The following document describes the manual OSD management steps for the deployment where we first found this: http://goo.gl/SPZGFC. Also, Miroslav Anashkin (<email address hidden>) has detailed context

[Oleksiy Molchanov (omolchanov)]

This issue doesn't affect 6.1

[Dmitriy Novakovskiy (dnovakovskiy)]

>>This issue doesn't affect 6.1

Why? What was changed in how Ceph is deployed in 6.1?

[Ryan Moe (rmoe)]

When Fuel deploys Ceph we set the GPT partition typecode (using sgdisk [0]) for OSD and journal partitions. Ceph installs udev rules [1] that will find all partitions containing these GUIDs and activate them as needed. If you're going to deploy new OSDs manually you'll probably want to set the partition GUIDs.

[0] https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cobbler/templates/scripts/pmanager.py#L885
[1] https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules

[Dmitriy Novakovskiy (dnovakovskiy)]

Ryan, do I understand correctly that you're describing how Ceph deployment is done in 6.1?

Miroslav, please verify that the approach that Ryan describes will help

[Ryan Moe (rmoe)]

This is how Fuel has deployed Ceph since the feature was added.

[Dmitriy Novakovskiy (dnovakovskiy)]

Then this needs to be reviewed by Miroslav. He first observed this problem at
customer installation

On Monday, February 9, 2015, Ryan Moe <email address hidden> wrote:

> This is how Fuel has deployed Ceph since the feature was added.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1416855
>
> Title:
> Ceph OSD disks are lost at node reboot
>
> Status in Fuel: OpenStack installer that works:
> In Progress
>
> Bug description:
> In one of customer deployments we faced the situation when host OS
> kernel would initialize multiple backplanes with disks in random order
> at boot time. This causes OSD disks that Fuel deployed w/ ceph-deploy
> utility to be lost from OSD after node reboot, since they were mounted
> via /dev/sdXXX pointers and these numbers become different at every
> boot.
>
> The solution is to mount OSD disks using UUID instead of sdXXX names.
> It's needs to be checked whether ceph-deploy utility can do it (so
> Fuel could just pass additional parameter) or a more sophisticated
> approach is needed to solve this problem.
>
> The following document describes the manual OSD management steps for
> the deployment where we first found this: http://goo.gl/SPZGFC. Also,
> Miroslav Anashkin (<email address hidden> <javascript:;>) has detailed
> context
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1416855/+subscriptions
>

--
---
Regards,

*Dmitriy Novakovskiy*
Sr. Sales Engineer, Mirantis EMEA

*Skype:* dmitriy.novakovskiy
*Operating from:* Ukraine

[Bogdan Dobrelya (bogdando)]

That was a version of Fuel for the reported issue?

[Sergii Golovatiuk (sgolovatiuk)]

Issue with OSD happens only when disks are added manually after deployment phase, during operation.

[Aleksandr Didenko (adidenko)]

Ryan and Sergii are right, such problem may affect only disks that were added manually after deployment. I've checked on 5.0.1 env I have and udev rules are in place and osd partitions have GUID set, so marking this invalid for 5.0.

[Ryan Moe (rmoe)]

This is invalid for 5.1 and 6.0 for the same reasons it's invalid for 6.1 and 5.0. We set the partition GUID during provisioning and the udev rules are present in both of those releases.

[Dmitry Borodaenko (angdraug)]

Please document the correct procedure to add disks to deployed OSD nodes in the Operations Guide. See the document linked from bug description for reference.

[Denis Klepikov (dklepikov)]

Draft "How to add OSD with mount by UDEV on reboot, with notes."

https://docs.google.com/a/mirantis.com/document/d/18gPSkw4Cg3cV5mHF-O3OxATMqPSatiIBwUw-l-uD1-k/edit?usp=sharing

Comments are welcome.

[Miroslav Anashkin (manashkin)]

Published bulletin:
https://online.mirantis.com/hubfs/Mirantis-Technical-Bulletin-5-Removing-Ceph-OSD-node.pdf?t=1427907150102

[Dmitriy Novakovskiy (dnovakovskiy)]

it seems to be dealing with another issue - removing OSD nodes from Fuel
UI, not OSD getting lost at reboot

On Wed, Apr 1, 2015 at 8:31 PM, Miroslav Anashkin <email address hidden>
wrote:

> Published bulletin:
>
> https://online.mirantis.com/hubfs/Mirantis-Technical-Bulletin-5-Removing-Ceph-OSD-node.pdf?t=1427907150102
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1416855
>
> Title:
> Ceph OSD disks are lost at node reboot
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
> Status in Fuel for OpenStack 5.0.x series:
> Won't Fix
> Status in Fuel for OpenStack 5.1.x series:
> Won't Fix
> Status in Fuel for OpenStack 6.0.x series:
> Confirmed
> Status in Fuel for OpenStack 6.1.x series:
> Confirmed
>
> Bug description:
> In one of customer deployments we faced the situation when host OS
> kernel would initialize multiple backplanes with disks in random order
> at boot time. This causes OSD disks that Fuel deployed w/ ceph-deploy
> utility to be lost from OSD after node reboot, since they were mounted
> via /dev/sdXXX pointers and these numbers become different at every
> boot.
>
> The solution is to mount OSD disks using UUID instead of sdXXX names.
> It's needs to be checked whether ceph-deploy utility can do it (so
> Fuel could just pass additional parameter) or a more sophisticated
> approach is needed to solve this problem.
>
> The following document describes the manual OSD management steps for
> the deployment where we first found this: http://goo.gl/SPZGFC. Also,
> Miroslav Anashkin (<email address hidden>) has detailed context
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1416855/+subscriptions
>

[Andrey Grebennikov (agrebennikov)]

This still doesn't help. In 6.0 right now we are experiencing this issue. We have 2 disks for journal, one for OS and 10 for OSDs. When node is bootstrapped, sda and sdb were assigned to be journals, sdc turned into OS, remaining disks became OSDs. In puppet log though I see /dev/sdl and /dev/sdk are the journal disks. If I reboot the node now, these last two disks will become /dev/sda and /dev/sdb and no OSD will be able to start anymore since they will not be able to find their journals.

[Igor Shishkin (teran)]

Moving to 6.1-updates since 6.1 is already released.

[Miroslav Anashkin (manashkin)]

Ceph-deploy 1.5.20 uses disk IDs to link journals to.
So, issue is fixed in 6.1 and higher versions.
Marked these versions as fix released, since we ship Ceph version with fix for these versions.

[Alexey Stupnikov (astupnikov)]

MOS 5.0, MOS 5.1 and MOS6.0 are no longer supported. Moving to Won't Fix.