Activity log for bug #1778266

Date Who What changed Old value New value Message
2018-06-22 19:40:32 Simon Monette bug added bug
2018-06-22 19:41:04 Simon Monette description When we deploy our Openstack bundle with a ceph cluster in it, most of the time the ceph charms will be in error. After looking at the logs, it seems ceph-radosgw tried to create its pool in ceph-mon, but when it does so before the other Openstack services can create theirs. Wich result in the following error: The logs on ceph-radosgw units are filled with the error messages: 2018-05-30 21:09:25.739133 7f48374f4e80 0 pidfile_write: ignore empty --pid-file 2018-05-30 21:09:25.748857 7f48374f4e80 -1 auth: unable to find a keyring on /etc/ceph/keyring.rados.gateway: (2) No such file or directory The file "/etc/ceph/keyring.rados.gateway" is indeed not present. Also on ceph-mon no other pools get to be created: # juju ssh ceph-mon/0 sudo ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 488G 488G 325M 0.07 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS default.rgw.buckets 1 0 0 154G 0 The only time the deployment worked, was because the ceph-radosgw units took more time then the other charms. We then tried to deploy the same bundle, but without adding the ceph-radosgw units to the machines and waited for the cinder pools to be created in in ceph-mon and then add the ceph-radosgw units to the machines. This time ceph-radosgw did not had any issue creating its keyring and all pools were visible in ceph-mon. # juju ssh ceph-mon/0 sudo ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 488G 487G 1223M 0.24 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cinder-backup 1 0 0 154G 0 cinder 2 19 0 154G 3 cinder-ceph 3 19 0 154G 3 glance 4 278M 0.18 154G 41 default.rgw.buckets 5 0 0 154G 0 default.rgw 6 0 0 154G 0 default.rgw.root 7 0 0 154G 0 default.rgw.control 8 0 0 154G 8 default.rgw.gc 9 0 0 154G 0 default.rgw.buckets.index 10 0 0 154G 0 default.rgw.buckets.extra 11 0 0 154G 0 default.log 12 0 0 154G 0 default.intent-log 13 0 0 154G 0 default.usage 14 0 0 154G 0 default.users 15 0 0 154G 0 default.users.email 16 0 0 154G 0 default.users.swift 17 0 0 154G 0 default.users.uid 18 0 0 154G 0 .rgw.root 19 1113 0 154G 4 default.rgw.meta 20 0 0 154G 0 default.rgw.log 21 0 0 154G 175 gnocchi 22 30624k 0.02 154G 3188 From there the Openstack deployment continued and worked without any issue. Maybe there is a race conditions in how ceph handle its relations. Environment: All the latest released charms on Xenial/Queen App Rev ceilometer 253 ceilometer-agent 244 ceph-mon 25 ceph-osd 262 ceph-radosgw 258 cinder 272 cinder-backup 13 cinder-ceph 233 glance 265 gnocchi 8 hacluster 46 heat 252 keystone 281 memcached 21 mysql 266 neutron-api 260 neutron-gateway 252 neutron-openvswitch 250 nova-cloud-controller 310 nova-compute 284 ntp 24 ntpmaster 7 openstack-dashboard 259 rabbitmq-server 74 When we deploy our Openstack bundle with a ceph cluster in it, most of the time the ceph charms will be in error. After looking at the logs, it seems ceph-radosgw tried to create its pool in ceph-mon, but when it does so before the other Openstack services can create theirs. Wich result in the following error: The logs on ceph-radosgw units are filled with the error messages: 2018-05-30 21:09:25.739133 7f48374f4e80 0 pidfile_write: ignore empty --pid-file 2018-05-30 21:09:25.748857 7f48374f4e80 -1 auth: unable to find a keyring on /etc/ceph/keyring.rados.gateway: (2) No such file or directory The file "/etc/ceph/keyring.rados.gateway" is indeed not present. Also on ceph-mon no other pools get to be created: # juju ssh ceph-mon/0 sudo ceph df GLOBAL:     SIZE AVAIL RAW USED %RAW USED     488G 488G 325M 0.07 POOLS:     NAME ID USED %USED MAX AVAIL OBJECTS     default.rgw.buckets 1 0 0 154G 0 The only time the deployment worked, was because the ceph-radosgw units took more time than the other charms. We then tried to deploy the same bundle, but without adding the ceph-radosgw units to the machines and waited for the cinder pools to be created in in ceph-mon and then add the ceph-radosgw units to the machines. This time ceph-radosgw did not had any issue creating its keyring and all pools were visible in ceph-mon. # juju ssh ceph-mon/0 sudo ceph df GLOBAL:     SIZE AVAIL RAW USED %RAW USED     488G 487G 1223M 0.24 POOLS:     NAME ID USED %USED MAX AVAIL OBJECTS     cinder-backup 1 0 0 154G 0     cinder 2 19 0 154G 3     cinder-ceph 3 19 0 154G 3     glance 4 278M 0.18 154G 41     default.rgw.buckets 5 0 0 154G 0     default.rgw 6 0 0 154G 0     default.rgw.root 7 0 0 154G 0     default.rgw.control 8 0 0 154G 8     default.rgw.gc 9 0 0 154G 0     default.rgw.buckets.index 10 0 0 154G 0     default.rgw.buckets.extra 11 0 0 154G 0     default.log 12 0 0 154G 0     default.intent-log 13 0 0 154G 0     default.usage 14 0 0 154G 0     default.users 15 0 0 154G 0     default.users.email 16 0 0 154G 0     default.users.swift 17 0 0 154G 0     default.users.uid 18 0 0 154G 0     .rgw.root 19 1113 0 154G 4     default.rgw.meta 20 0 0 154G 0     default.rgw.log 21 0 0 154G 175     gnocchi 22 30624k 0.02 154G 3188 From there the Openstack deployment continued and worked without any issue. Maybe there is a race conditions in how ceph handle its relations. Environment: All the latest released charms on Xenial/Queen App Rev ceilometer 253 ceilometer-agent 244 ceph-mon 25 ceph-osd 262 ceph-radosgw 258 cinder 272 cinder-backup 13 cinder-ceph 233 glance 265 gnocchi 8 hacluster 46 heat 252 keystone 281 memcached 21 mysql 266 neutron-api 260 neutron-gateway 252 neutron-openvswitch 250 nova-cloud-controller 310 nova-compute 284 ntp 24 ntpmaster 7 openstack-dashboard 259 rabbitmq-server 74
2018-07-20 09:43:30 Dominique Poulain bug added subscriber Dominique Poulain
2018-10-01 17:58:31 Narinder Gupta information type Public Private
2018-10-01 18:00:17 Narinder Gupta information type Private Public
2018-10-01 18:02:00 Dmitrii Shcherbakov tags cpe-onsite
2018-10-01 18:02:07 Dmitrii Shcherbakov bug added subscriber Dmitrii Shcherbakov
2018-10-01 18:04:55 Dmitrii Shcherbakov charm-ceph-radosgw: status New Triaged
2018-10-01 18:04:57 Dmitrii Shcherbakov charm-ceph-radosgw: importance Undecided High
2018-10-03 15:54:48 Narinder Gupta bug added subscriber Canonical Field High
2018-10-03 16:23:06 Ryan Beisner charm-ceph-radosgw: status Triaged Incomplete
2018-10-03 16:32:58 Ryan Beisner charm-ceph-radosgw: milestone 18.11
2018-10-03 16:33:07 Ryan Beisner charm-ceph-radosgw: assignee Ryan Beisner (1chb1n)
2018-10-04 15:13:14 Ryan Beisner marked as duplicate 1774279