And I also see the Unexpected error in broker-rsp-ceph-radosgw-5:
ubuntu@hevelius:~$ juju debug-log -i ceph-radosgw/5 --replay | grep -i "Request already sent" unit-ceph-radosgw-5: 17:26:56 DEBUG unit.ceph-radosgw/5.juju-log mon:65: Request already sent but not complete, not sending new request
ubuntu@hevelius:~$ juju run --unit ceph-radosgw/5 'relation-get -r mon:65 - ceph-mon/0' auth: cephx ceph-public-address: 10.0.3.24 egress-subnets: 192.168.210.135/32 fsid: 4e0ece9e-a1d5-11ea-8976-00163e2267d7 ingress-address: 192.168.210.135 private-address: 192.168.210.135 radosgw_key: AQB6GNVelHpAJRAAGFQJyxi1hpOnaeuMdstXgQ== rgw.juju-10aa60-3-lxd-5_key: AQAEOtVe1YbsDBAAqQHMZNpZUUZ+QJFDgaEyJQ==
ubuntu@hevelius:~$ juju run --unit ceph-radosgw/5 'relation-get -r mon:65 - ceph-mon/1' auth: cephx ceph-public-address: 10.0.3.25 egress-subnets: 192.168.210.137/32 fsid: 4e0ece9e-a1d5-11ea-8976-00163e2267d7 ingress-address: 192.168.210.137 private-address: 192.168.210.137 radosgw_key: AQB6GNVelHpAJRAAGFQJyxi1hpOnaeuMdstXgQ== rgw.juju-10aa60-3-lxd-5_key: AQAEOtVe1YbsDBAAqQHMZNpZUUZ+QJFDgaEyJQ==
ubuntu@hevelius:~$ juju run --unit ceph-radosgw/5 'relation-get -r mon:65 - ceph-mon/2' auth: cephx broker-rsp-ceph-radosgw-5: '{"exit-code": 1, "stderr": "Unexpected error occurred while processing requests: {''api-version'': 1, ''ops'': [{''op'': ''create-pool'', ''name'': ''default.rgw.buckets.data'', ''replicas'': 3, ''pg_num'': None, ''weight'': 20, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.control'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.data.root'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.gc'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.log'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.intent-log'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.meta'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.usage'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.users.keys'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.users.email'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.users.swift'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.users.uid'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.buckets.extra'', ''replicas'': 3, ''pg_num'': None, ''weight'': 1.0, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.rgw.buckets.index'', ''replicas'': 3, ''pg_num'': None, ''weight'': 3.0, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''.rgw.root'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-namespace'': None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}], ''request-id'': ''d792628a-a42c-11ea-94c6-00163efd72ee''}"}' ceph-public-address: 10.0.3.21 egress-subnets: 192.168.210.126/32 fsid: 4e0ece9e-a1d5-11ea-8976-00163e2267d7 ingress-address: 192.168.210.126 private-address: 192.168.210.126 radosgw_key: AQB6GNVelHpAJRAAGFQJyxi1hpOnaeuMdstXgQ== rgw.juju-10aa60-3-lxd-5_key: AQAEOtVe1YbsDBAAqQHMZNpZUUZ+QJFDgaEyJQ==
And I also see the Unexpected error in broker- rsp-ceph- radosgw- 5:
ubuntu@hevelius:~$ juju debug-log -i ceph-radosgw/5 --replay | grep -i "Request already sent" radosgw- 5: 17:26:56 DEBUG unit.ceph- radosgw/ 5.juju- log mon:65: Request already sent but not complete, not sending new request
unit-ceph-
ubuntu@hevelius:~$ juju run --unit ceph-radosgw/5 'relation-get -r mon:65 - ceph-mon/0' address: 10.0.3.24 a1d5-11ea- 8976-00163e2267 d7 AGFQJyxi1hpOnae uMdstXgQ= = 10aa60- 3-lxd-5_ key: AQAEOtVe1YbsDBA AqQHMZNpZUUZ+ QJFDgaEyJQ= =
auth: cephx
ceph-public-
egress-subnets: 192.168.210.135/32
fsid: 4e0ece9e-
ingress-address: 192.168.210.135
private-address: 192.168.210.135
radosgw_key: AQB6GNVelHpAJRA
rgw.juju-
ubuntu@hevelius:~$ juju run --unit ceph-radosgw/5 'relation-get -r mon:65 - ceph-mon/1' address: 10.0.3.25 a1d5-11ea- 8976-00163e2267 d7 AGFQJyxi1hpOnae uMdstXgQ= = 10aa60- 3-lxd-5_ key: AQAEOtVe1YbsDBA AqQHMZNpZUUZ+ QJFDgaEyJQ= =
auth: cephx
ceph-public-
egress-subnets: 192.168.210.137/32
fsid: 4e0ece9e-
ingress-address: 192.168.210.137
private-address: 192.168.210.137
radosgw_key: AQB6GNVelHpAJRA
rgw.juju-
ubuntu@hevelius:~$ juju run --unit ceph-radosgw/5 'relation-get -r mon:65 - ceph-mon/2' rsp-ceph- radosgw- 5: '{"exit-code": 1, "stderr": "Unexpected error occurred rgw.buckets. data'', ''replicas'': 3, ''pg_num'': None, ''weight'': namespace' ': None, ''app-name'': ''rgw'', ''max-bytes'': rgw.control' ', namespace' ': rgw.data. root'', ''replicas'': 3, ''pg_num'': namespace' ': None, ''app-name'': namespace' ': None, ''app-name'': ''rgw'', ''max-bytes'': rgw.log' ', namespace' ': rgw.intent- log'', ''replicas'': 3, ''pg_num'': namespace' ': None, ''app-name'': rgw.meta' ', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, namespace' ': None, ''app-name'': ''rgw'', ''max-bytes'': rgw.usage' ', namespace' ': rgw.users. keys'', ''replicas'': 3, ''pg_num'': namespace' ': None, ''app-name'': rgw.users. email'' , ''replicas'': 3, ''pg_num'': None, ''weight'': namespace' ': None, ''app-name'': ''rgw'', ''max-bytes'': rgw.users. swift'' , namespace' ': rgw.users. uid'', ''replicas'': 3, ''pg_num'': namespace' ': None, ''app-name'': rgw.buckets. extra'' , ''replicas'': 3, ''pg_num'': None, ''weight'': namespace' ': None, ''app-name'': ''rgw'', ''max-bytes'': rgw.buckets. index'' , namespace' ': namespace' ': None, ''app-name'': ''rgw'', ''max-bytes'': a42c-11ea- 94c6-00163efd72 ee''}"} ' address: 10.0.3.21 a1d5-11ea- 8976-00163e2267 d7 AGFQJyxi1hpOnae uMdstXgQ= = 10aa60- 3-lxd-5_ key: AQAEOtVe1YbsDBA AqQHMZNpZUUZ+ QJFDgaEyJQ= =
auth: cephx
broker-
while processing requests: {''api-version'': 1, ''ops'': [{''op'': ''create-pool'',
''name'': ''default.
20, ''group'': ''objects'', ''group-
None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.
''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-
None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'':
''create-pool'', ''name'': ''default.
None, ''weight'': 0.1, ''group'': ''objects'', ''group-
''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'',
''name'': ''default.rgw.gc'', ''replicas'': 3, ''pg_num'': None, ''weight'': 0.1,
''group'': ''objects'', ''group-
None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.
''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-
None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'':
''create-pool'', ''name'': ''default.
None, ''weight'': 0.1, ''group'': ''objects'', ''group-
''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'',
''name'': ''default.
''group'': ''objects'', ''group-
None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.
''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-
None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'':
''create-pool'', ''name'': ''default.
None, ''weight'': 0.1, ''group'': ''objects'', ''group-
''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'',
''name'': ''default.
0.1, ''group'': ''objects'', ''group-
None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.
''replicas'': 3, ''pg_num'': None, ''weight'': 0.1, ''group'': ''objects'', ''group-
None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'':
''create-pool'', ''name'': ''default.
None, ''weight'': 0.1, ''group'': ''objects'', ''group-
''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'': ''create-pool'',
''name'': ''default.
1.0, ''group'': ''objects'', ''group-
None, ''max-objects'': None}, {''op'': ''create-pool'', ''name'': ''default.
''replicas'': 3, ''pg_num'': None, ''weight'': 3.0, ''group'': ''objects'', ''group-
None, ''app-name'': ''rgw'', ''max-bytes'': None, ''max-objects'': None}, {''op'':
''create-pool'', ''name'': ''.rgw.root'', ''replicas'': 3, ''pg_num'': None, ''weight'':
0.1, ''group'': ''objects'', ''group-
None, ''max-objects'': None}], ''request-id'': ''d792628a-
ceph-public-
egress-subnets: 192.168.210.126/32
fsid: 4e0ece9e-
ingress-address: 192.168.210.126
private-address: 192.168.210.126
radosgw_key: AQB6GNVelHpAJRA
rgw.juju-