ceph: default crush rule does not suit multi-OSD deployments

[Paul Collins]

Version: 0.48.2-0ubuntu2~cloud0

Our Ceph deployments typically involve multiple OSDs per host with no disk redundancy. However the default crush rules appears to distribute by OSD, not by host, which I believe will not prevent replicas from landing on the same host.

I've been working around this by updating the crush rules as follows and installing the resulting crushmap in the cluster, but since we aim for fully automated deployment (using Juju and MaaS) this is suboptimal.

--- crushmap.txt 2013-01-10 20:33:21.265809301 +0000
+++ crushmap.new 2013-01-10 20:32:49.496745778 +0000
@@ -104,7 +104,7 @@
        min_size 1
        max_size 10
        step take default
- step choose firstn 0 type osd
+ step chooseleaf firstn 0 type host
        step emit
 }
 rule metadata {
@@ -113,7 +113,7 @@
        min_size 1
        max_size 10
        step take default
- step choose firstn 0 type osd
+ step chooseleaf firstn 0 type host
        step emit
 }
 rule rbd {
@@ -122,7 +122,7 @@
        min_size 1
        max_size 10
        step take default
- step choose firstn 0 type osd
+ step chooseleaf firstn 0 type host
        step emit
 }

[Paul Collins (pjdc)]

This has been fixed on upstream's master branch by commit c236a51a8040508ee893e4c64b206e40f9459a62 and cherry-picked to the bobtail branch as 6008b1d8e4587d5a3aea60684b1d871401496942. The change does not seem to have been applied to argonaut.

[Sage Weil (sage-newdream)]

Is it important that this is backported to argonaut, given bobtail will be in 13.04?

[Paul Collins (pjdc)]

From my point of view, probably not very. We (= Canonical IS) are running 12.04 LTS plus packages from the Ubuntu Cloud Archive. I don't believe we'll do many more folsom+argonaut deployments before grizzly+bobtail arrives, and in any case it's sufficiently well documented internally that it's not a big problem for us.