Ceph pools requested over relations are not created (and failed without any notification)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceph Monitor Charm |
Triaged
|
Medium
|
Unassigned | ||
charms.ceph |
Triaged
|
Medium
|
Unassigned |
Bug Description
bionic-train, latest charms revs from charmstore.
juju status: https:/
full bundle: https:/
When I tried to upload an image, Glance returned http500:
ubuntu@
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
(verbose: https:/
Here's an error from glance.log:
2020-04-10 09:36:28.841 81289 DEBUG glance_
707e47] Store glance_
thon3/dist-
2020-04-10 09:36:28.841 81289 DEBUG glance_store.driver [req-278632e9-
] Late loading location class glance_
2020-04-10 09:36:28.841 81289 DEBUG glance_
47] Registering scheme rbd with {'store': <glance_
/usr/lib/
2020-04-10 09:36:28.858 81289 ERROR glance.
07e47] Failed to upload image data due to internal error: rados.ObjectNot
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi [req-278632e9-
Caught error: [errno 2] error opening pool 'b'glance'': rados.ObjectNot
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi Traceback (most recent call last):
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi request, **action_args)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi return method(*args, **kwargs)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi self._restore(
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi self.force_
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi six.reraise(
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi raise value
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi image.set_
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi self.base.
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi _send_notificat
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi self.force_
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi six.reraise(
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi raise value
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi self.repo.
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi return self.image.
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi self.image.
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi verifier=verifier)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi metadata_dict) = store_add_
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "/usr/lib/
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi with conn.open_
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "rados.pyx", line 516, in rados.requires.
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi File "rados.pyx", line 1203, in rados.Rados.
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi rados.ObjectNot
2020-04-10 09:36:28.898 81289 ERROR glance.common.wsgi
2020-04-10 09:36:28.941 81289 INFO eventlet.
And there are no pool created by glance charm:
ubuntu@
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 108 GiB 104 GiB 5.1 MiB 4.0 GiB 3.71
TOTAL 108 GiB 104 GiB 5.1 MiB 4.0 GiB 3.71
POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
default.
default.
default.
default.rgw.gc 4 0 B 0 0 B 0 33 GiB
default.rgw.log 5 0 B 175 0 B 0 33 GiB
default.
default.
default.
default.
default.
default.
default.
default.
default.
.rgw.root 15 1.2 KiB 4 48 KiB 0 33 GiB
Connection to 172.27.85.9 closed.
ubuntu@
(compared with another deployment, there are some pools missing - glance, gnocchi etc... https:/
Looking into ceph-mon unit log, I found the following (full log: http://
unit-ceph-mon-0: 09:10:48 DEBUG unit.ceph-
unit-ceph-mon-0: 09:10:48 ERROR unit.ceph-
unit-ceph-mon-0: 09:10:48 ERROR unit.ceph-
Changed in charm-ceph-mon: | |
assignee: | nobody → Rodrigo Barbieri (rodrigo-barbieri2010) |
Changed in charm-ceph-mon: | |
assignee: | Rodrigo Barbieri (rodrigo-barbieri2010) → nobody |
Changed in charms.ceph: | |
status: | New → Triaged |
importance: | Undecided → Medium |
Problem got workarounded with customizing ceph-mon config and restarting MON daemons:
$ juju config ceph-mon config-flags
{'global': {'mon max pg per osd': 100000}}