Deploying ceph-mon and ceph-osd on same VMs (no isolation) doesn't work

[Aymen Frikha]

Hi,

I've tried to deploy ceph cluster on VMs. I deployed ceph-mon and ceph-osd directly into the VM (without lxc for ceph-mon). ceph-osd service is stayed in blocked status "No block devices detected using current configuration".
I'm running these versions of the charm:
cs:trusty/ceph-osd-235
cs:trusty/ceph-mon-1

This is the content of /etc/ceph/ceph.conf:
[global]

auth cluster required = cephx
auth service required = cephx
auth client required = cephx

keyring = /etc/ceph/$cluster.$name.keyring
mon host = 192.168.11.202:6789 192.168.11.203:6789 192.168.11.204:6789
fsid = 0e3be3da-0100-434d-8707-1b159f5f9832

log to syslog = false
err to syslog = false
clog to syslog = false
mon cluster log to syslog = false
debug mon = 1/5
debug osd = 1/5
public network = 192.168.11.0/24
cluster network = 192.168.144.0/24

public addr = 192.168.11.203

cluster addr = 192.168.144.203

[mon]
keyring = /var/lib/ceph/mon/$cluster-$id/keyring

[mds]
keyring = /var/lib/ceph/mds/$cluster-$id/keyring

This is the content of /var/log/upstart/ceph-osd-ceph_1.log:

 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
2016-07-20 11:28:32.191329 7f0d669ff800 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2016-07-20 11:28:32.207789 7f0d669ff800 -1 ** ERROR: osd init failed: (2) No such file or directory
create-or-move updated item name 'osd.1' weight 0.01 at location {host=ceph2,root=default} to crush map
starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
2016-07-20 11:28:32.843634 7f24b244e800 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2016-07-20 11:28:32.859870 7f24b244e800 -1 ** ERROR: osd init failed: (2) No such file or directory

I think there is a section for osd that is missed in /etc/ceph/ceph.conf.

[James Page (james-page)]

Actually this is by design; you should place ceph-mon on separate machines, or in LXD/LXC containers using Juju.