Puppet should exit with error if disk activate fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
puppet-ceph |
Fix Released
|
Undecided
|
John Fulton |
Bug Description
Ceph osds get created only for the first deployment. On any subsequent deployments the OSDs fail to get created:
Steps to reproduce:
First deployment:
source ~/stackrc
export THT='/home/
openstack overcloud deploy --templates $THT \
-e $THT/environmen
-e $THT/environmen
-e ~/templates/
-e $THT/environmen
-e $THT/environmen
-e $THT/environmen
-e ~/templates/
--control-scale 3 \
--control-flavor controller \
--compute-scale 3 \
--compute-flavor compute \
--ntp-server clock.redhat.com \
--libvirt-type qemu
Make sure that the deployment was successful and OSDs got created:
[stack@undercloud ~]$ cat templates/
parameter_defaults:
ExtraConfig:
ceph:
'/dev/vdb': {}
'/dev/vdc': {}
[root@overcloud
# id weight type name up/down reweight
-1 0.1199 root default
-2 0.03998 host overcloud-
0 0.01999 osd.0 up 1
3 0.01999 osd.3 up 1
-3 0.03998 host overcloud-
1 0.01999 osd.1 up 1
4 0.01999 osd.4 up 1
-4 0.03998 host overcloud-
2 0.01999 osd.2 up 1
5 0.01999 osd.5 up 1
Delete existing deployment
Redeploy using the initial deploy command.
Check OSDs:
[root@overcloud
# id weight type name up/down reweight
-1 0 root default
We can see the following errors in the os-collect-config journal:
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
As a workaround we can erase the GPT tables of the OSD disks before redeploying:
sgdisk --zap /dev/vdb
sgdisk --zap /dev/vdc
Changed in puppet-ceph: | |
status: | New → Confirmed |
Changed in puppet-ceph: | |
assignee: | nobody → John Fulton (jfulton-org) |
Changed in puppet-ceph: | |
status: | Confirmed → In Progress |
Speaking with gfidente about it, this is expected behavior as the disks are not erased when deleting the deployment so the 2nd deployment will fail when running disk activate.
Nevertheless the deployment completed fine even though disk activate failed so I'm turning the bug against this issue as the deployment should fail if disk activate fails.
Below is he full log:
Notice: /Stage[ main]/Snmp/ Service[ snmpd]/ ensure: ensure changed 'stopped' to 'running' main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + test -b /dev/vdc main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + test -b /dev/vdc main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + test -b /dev/vdc1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + test -f /usr/lib/ udev/rules. d/95-ceph- osd.rules. disabled main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + test -b /dev/vdc1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + ceph-disk activate /dev/vdc1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: === osd.5 === main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: 2016-07-20 08:49:41.150384 7f2f29a8c700 0 librados: osd.5 authentication error (1) Operation not permitted main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: Error connecting to cluster: PermissionError main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.5 --keyring= /var/lib/ ceph/osd/ ceph-5/ keyring osd crush create-or-move -- 5 0.02 host=overcloud- novacompute- 0 root=default' main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: ceph-disk: Error: ceph osd start failed: Command '['/usr/ sbin/service' , 'ceph', '--cluster', 'ceph', 'start', 'osd.5']' returned non-zero exit status 1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: + true main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdc]/Exec[ ceph-osd- activate- /dev/vdc] /returns: executed successfully main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /dev/vdb] /returns: + test -b /dev/vdb main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /dev/vdb] /returns: + test -b /dev/vdb main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /dev/vdb] /returns: + test -b /dev/vdb1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /dev/vdb] /returns: + test -f /usr/lib/ udev/rules. d/95-ceph- osd.rules. disabled main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /dev/vdb] /returns: + test -b /dev/vdb1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /dev/vdb] /returns: + ceph-disk activate /dev/vdb1 main]/Ceph: :Osds/Ceph: :Osd[/dev/ vdb]/Exec[ ceph-osd- activate- /...
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[