All the ceph related functionality and config are gone after server reboot

[Imran]

I am trying to install devstack(liberty)
by clonning as:
git clone https://github.com/openstack-dev/devstack.git -b stable/liberty

Then install ceph by using the following plugin in localrc file:
# Enable ceph DevStack plugin
enable_plugin devstack-plugin-ceph git://git.openstack.org/openstack/devstack-plugin-ceph

And it comes up fine
The problem is that when I reboot the server, I lost all ceph configuration.

All my ceph commands stop working and I am getting the following errors:

adminx@cephcontrail:~$ sudo ceph status
2016-03-17 15:55:55.489590 7fa34c7c8700 0 -- :/3530400219 >> 192.168.57.64:6789/0 pipe(0x7fa34805d050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fa348059c50).fault
2016-03-17 15:55:58.489293 7fa34c6c7700 0 -- :/3530400219 >> 192.168.57.64:6789/0 pipe(0x7fa33c000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fa33c004ef0).fault
^CError connecting to cluster: InterruptedOrTimeoutError
adminx@cephcontrail:~$
adminx@cephcontrail:~$
adminx@cephcontrail:~$
adminx@cephcontrail:~$ sudo ceph mon stat
2016-03-17 15:56:05.009688 7fa050226700 0 -- :/3529225905 >> 192.168.57.64:6789/0 pipe(0x7fa04c05d050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fa04c059c90).fault
2016-03-17 15:56:08.010524 7fa050125700 0 -- :/3529225905 >> 192.168.57.64:6789/0 pipe(0x7fa040000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fa040004ef0).fault
^CError connecting to cluster: InterruptedOrTimeoutEr

The following file system disappears from mount
adminx@cephos2:~/devstack$ sudo mount | grep ceph

Before reboot, I was getting the following output

/var/lib/ceph/drives/images/ceph.img on /var/lib/ceph type xfs (rw,noatime,nodiratime,nobarrier,logbufs=8)

And all the following ceph monitor and osd related files disappear after the reboot:

Before reboot, I was getting the following output:

adminx@cephos2:~/devstack$ ls -lrt /var/lib/ceph/
total 0
drwxr-xr-x 2 root root 6 Mar 16 12:01 radosgw
drwxr-xr-x 2 root root 6 Mar 16 12:01 mds
drwxr-xr-x 2 root root 32 Mar 16 12:01 tmp
drwxr-xr-x 3 root root 25 Mar 16 12:01 mon
drwxr-xr-x 2 root root 25 Mar 16 12:01 bootstrap-osd
drwxr-xr-x 2 root root 25 Mar 16 12:01 bootstrap-rgw
drwxr-xr-x 2 root root 25 Mar 16 12:01 bootstrap-mds
drwxr-xr-x 3 root root 19 Mar 16 12:01 osd

adminx@cephos2:~/devstack$ ls -lrt /var/lib/ceph/mon/ceph-cephos2/
total 4
-rw-r--r-- 1 root root 77 Mar 16 12:01 keyring
-rw-r--r-- 1 root root 0 Mar 16 12:01 upstart
drwxr-xr-x 2 root root 128 Mar 16 12:01 store.db

adminx@cephos2:~$ ls -lrt /var/lib/ceph/osd
total 0
drwxr-xr-x 3 root root 163 Mar 16 12:01 ceph-0
adminx@cephos2:~$
adminx@cephos2:~$ ls -lrt /var/lib/ceph/mon
total 0
drwxr-xr-x 3 root root 49 Mar 16 12:01 ceph-cephos2
adminx@cephos2:~$
adminx@cephos2:~$
adminx@cephos2:~$ ls -lrt /var/lib/ceph/mds/
total 0
adminx@cephos2:~$ ls -lrt /var/lib/ceph/osd/ceph-0/
total 102436
-rw-r--r-- 1 root root 53 Mar 16 12:01 superblock
-rw-r--r-- 1 root root 4 Mar 16 12:01 store_version
-rw-r--r-- 1 root root 37 Mar 16 12:01 fsid
-rw-r--r-- 1 root root 2 Mar 16 12:01 whoami
-rw-r--r-- 1 root root 6 Mar 16 12:01 ready
-rw-r--r-- 1 root root 21 Mar 16 12:01 magic
-rw-r--r-- 1 root root 37 Mar 16 12:01 ceph_fsid
-rw-r--r-- 1 root root 56 Mar 16 12:01 keyring
-rw-r--r-- 1 root root 0 Mar 16 12:01 upstart
drwxr-xr-x 92 root root 4096 Mar 16 12:31 current
-rw-r--r-- 1 root root 104857600 Mar 17 15:41 journal
adminx@cephos2:~$
adminx@cephos2:~$
adminx@cephos2:~$ ls -lrt /var/lib/ceph/mon/ceph-cephos2/
total 4
-rw-r--r-- 1 root root 77 Mar 16 12:01 keyring
-rw-r--r-- 1 root root 0 Mar 16 12:01 upstart
drwxr-xr-x 2 root root 230 Mar 17 16:00 store.db

After reboot, all the above files are gone

It seems like that I need to modify my /etc/fstab file to make it persistence and make some other ceph related changes so that it stays after the reboot

So it seems like a issue that after the reboot, everything related to ceph is gone

[melanie witt (melwitt)]

Does anyone know a workaround for this? I tried "sudo apt-get purge ceph*" and it wasn't enough. It instead got hung up on something related to cinder secrets.

[melanie witt (melwitt)]

Update: As a workaround, I'm able to stack.sh after a reboot by doing:

sudo virsh secret-list
sudo virsh secret-undefine <uuid>
sudo apt-get purge ceph*
./stack.sh