luminous: Couldn't init storage provider (RADOS)

[Josh Kirkpatrick]

I am unable to initialize the radosgw service in the ceph-radosgw charm. For versions radosgw/257 and /256. I searched and appears to be an issue with rados. See bug report >> http://tracker.ceph.com/issues/23349.

It states the but is planned to be fixed for 12.2.5, however current releaased charms has only up to 12.2.4

  ceph-radosgw:
    charm: cs:ceph-radosgw-256
    num_units: 1
    options:
      os-admin-network: 172.x.x.0/24
      os-internal-network: 172.x.x.0/24
      os-public-network: 10.y.y.y/27
      source: 'cloud:xenial-queens'
    constraints: 'spaces=default,os-internal-net'
    expose: true
    annotations:
      gui-x: '1000'
      gui-y: '250'
    to:
      - '0'
Relations:
    - - 'ceph-radosgw:mon'
    - 'ceph-mon:radosgw'
  - - 'ceph-radosgw:identity-service'

root@juju-fe4a6f-0-lxd-2:/var/log/apache2# ceph --version
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)

Please note I did perform the following :
    juju ssh ceph/0 \
      'sudo radosgw-admin user create --uid="ubuntu" --display-name="Ubuntu Ceph"'

user@us-a-reg-maas:~/openstack-base$ juju ssh ceph-mon/0
ubuntu@juju-fe4a6f-1-lxd-0:~$ sudo radosgw-admin user list
[
    "ubuntu"
]

I am attaching logs that I believe are useful and necessary.

[Josh Kirkpatrick (rockfreak101)]

Added glance to impacted projects, because if ceph-radosgw is deployed with glance, while glance uses ceph, it will appear that glance times out, but in fact its that the radosgw is unreachable.

[Josh Kirkpatrick (rockfreak101)]

I found the issue myself. the glance charm requires either additional routing provided, or has the storage networks on the charm as well. Once adding the storage networks to the glance charm, I was able to connect to radosgw.

[Dmitrii Shcherbakov (dmitriis)]

It's a subtle requirement but you are right that glance <-> ceph connectivity is required as glance stores images in Ceph and, therefore, needs access to both monitors and OSDs.

In general, any ceph client needs the same type of network connectivity whether it's QEMU (nova), glance, cinder, rados-gateway, gnocchi or something else.

You could place a constraint or add a binding to glance (ceph: *ceph-access-space):

variables:
  ceph-access-constr: &ceph-access-constr spaces=ceph-access-space

#...
  glance:
    charm: cs:xenial/glance
    constraints: *ceph-access-constr # <-
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space

[Dmitrii Shcherbakov (dmitriis)]

Josh,

I'll mark this bug as invalid if you don't mind.

[Josh Kirkpatrick (rockfreak101)]

I guess that's fine. Maybe there is an opportunity for me though
documentation in one of the charms?

But correct, this is not a bug, you may close.

On Mon, Jun 18, 2018, 12:20 Dmitrii Shcherbakov <email address hidden>
wrote:

> Josh,
>
> I'll mark this bug as invalid if you don't mind.
>
> ** Changed in: charm-ceph-radosgw
> Status: New => Invalid
>
> ** Changed in: charm-glance
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1775623
>
> Title:
> luminous: Couldn't init storage provider (RADOS)
>
> Status in OpenStack ceph-radosgw charm:
> Invalid
> Status in OpenStack glance charm:
> Invalid
>
> Bug description:
> I am unable to initialize the radosgw service in the ceph-radosgw
> charm. For versions radosgw/257 and /256. I searched and appears to be
> an issue with rados. See bug report >>
> http://tracker.ceph.com/issues/23349.
>
> It states the but is planned to be fixed for 12.2.5, however current
> releaased charms has only up to 12.2.4
>
> ceph-radosgw:
> charm: cs:ceph-radosgw-256
> num_units: 1
> options:
> os-admin-network: 172.x.x.0/24
> os-internal-network: 172.x.x.0/24
> os-public-network: 10.y.y.y/27
> source: 'cloud:xenial-queens'
> constraints: 'spaces=default,os-internal-net'
> expose: true
> annotations:
> gui-x: '1000'
> gui-y: '250'
> to:
> - '0'
> Relations:
> - - 'ceph-radosgw:mon'
> - 'ceph-mon:radosgw'
> - - 'ceph-radosgw:identity-service'
>
>
> root@juju-fe4a6f-0-lxd-2:/var/log/apache2# ceph --version
> ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous
> (stable)
>
>
> Please note I did perform the following :
> juju ssh ceph/0 \
> 'sudo radosgw-admin user create --uid="ubuntu"
> --display-name="Ubuntu Ceph"'
>
> user@us-a-reg-maas:~/openstack-base$ juju ssh ceph-mon/0
> ubuntu@juju-fe4a6f-1-lxd-0:~$ sudo radosgw-admin user list
> [
> "ubuntu"
> ]
>
>
> I am attaching logs that I believe are useful and necessary.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-radosgw/+bug/1775623/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/576230

[Dmitrii Shcherbakov (dmitriis)]

Josh,

I think you're right, it is quite frustrating to debug.

I had to do this at least once myself - raised a review to document the requirement.

[Josh Kirkpatrick (rockfreak101)]

Fantastic

On Mon, Jun 18, 2018, 13:50 Dmitrii Shcherbakov <email address hidden>
wrote:

> Josh,
>
> I think you're right, it is quite frustrating to debug.
>
> I had to do this at least once myself - raised a review to document the
> requirement.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1775623
>
> Title:
> luminous: Couldn't init storage provider (RADOS)
>
> Status in OpenStack ceph-radosgw charm:
> Invalid
> Status in OpenStack glance charm:
> Invalid
>
> Bug description:
> I am unable to initialize the radosgw service in the ceph-radosgw
> charm. For versions radosgw/257 and /256. I searched and appears to be
> an issue with rados. See bug report >>
> http://tracker.ceph.com/issues/23349.
>
> It states the but is planned to be fixed for 12.2.5, however current
> releaased charms has only up to 12.2.4
>
> ceph-radosgw:
> charm: cs:ceph-radosgw-256
> num_units: 1
> options:
> os-admin-network: 172.x.x.0/24
> os-internal-network: 172.x.x.0/24
> os-public-network: 10.y.y.y/27
> source: 'cloud:xenial-queens'
> constraints: 'spaces=default,os-internal-net'
> expose: true
> annotations:
> gui-x: '1000'
> gui-y: '250'
> to:
> - '0'
> Relations:
> - - 'ceph-radosgw:mon'
> - 'ceph-mon:radosgw'
> - - 'ceph-radosgw:identity-service'
>
>
> root@juju-fe4a6f-0-lxd-2:/var/log/apache2# ceph --version
> ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous
> (stable)
>
>
> Please note I did perform the following :
> juju ssh ceph/0 \
> 'sudo radosgw-admin user create --uid="ubuntu"
> --display-name="Ubuntu Ceph"'
>
> user@us-a-reg-maas:~/openstack-base$ juju ssh ceph-mon/0
> ubuntu@juju-fe4a6f-1-lxd-0:~$ sudo radosgw-admin user list
> [
> "ubuntu"
> ]
>
>
> I am attaching logs that I believe are useful and necessary.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-radosgw/+bug/1775623/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/576230
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=64180d831d68d12f301fb374da22a1fea9c87058
Submitter: Zuul
Branch: master

commit 64180d831d68d12f301fb374da22a1fea9c87058
Author: Dmitrii Shcherbakov <email address hidden>
Date: Mon Jun 18 21:38:34 2018 +0300

    add ceph ip connectivity requirement clarification

    Change-Id: I0f6159309260f6af3968729bb57b382095aec813
    Related-Bug: #1775623