ceph-radosgw tries to use keystone PKI when it was removed in Pike

Bug #1758982 reported by David Ames
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
Fix Released
Medium
Hemanth Nakkina
OpenStack Keystone Charm
Invalid
Undecided
Unassigned

Bug Description

Somewhat related to Bug#1718467.
The ceph-radosgw charm [0] and possibly radosgw itself attempt to use keystone as a PKI. PKI is removed in keystone at Pike. This leads to errors in keystone.log like the following:

(keystoneclient.common.cms): 2018-03-09 04:01:44,271 ERROR Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_s
etup"
(keystone.common.wsgi): 2018-03-09 04:01:44,272 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 228, in __call__
    result = method(req, **params)
  File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 94, in inner
    return f(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 350, in revocation_list
    CONF.signing.keyfile)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
    signing_key_file_name, message_digest=message_digest)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
    raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3

This does not affect keystone performance but does lead to confusion when reading the logs.

[0] https://github.com/openstack/charm-ceph-radosgw/blob/master/hooks/utils.py#L497

Revision history for this message
David Ames (thedac) wrote :

Although this bug is not in the keystone charm, adding it here and marking it invalid, as the keystone charm is the most likely place others will look for this bug.

Changed in charm-ceph-radosgw:
status: New → Confirmed
importance: Undecided → Medium
milestone: none → 18.05
Changed in charm-keystone:
status: New → Invalid
James Page (james-page)
Changed in charm-ceph-radosgw:
milestone: 18.05 → 18.08
James Page (james-page)
Changed in charm-ceph-radosgw:
milestone: 18.08 → 18.11
James Page (james-page)
Changed in charm-ceph-radosgw:
milestone: 18.11 → 19.04
David Ames (thedac)
Changed in charm-ceph-radosgw:
milestone: 19.04 → 19.07
David Ames (thedac)
Changed in charm-ceph-radosgw:
milestone: 19.07 → 19.10
David Ames (thedac)
Changed in charm-ceph-radosgw:
milestone: 19.10 → 20.01
James Page (james-page)
Changed in charm-ceph-radosgw:
milestone: 20.01 → 20.05
David Ames (thedac)
Changed in charm-ceph-radosgw:
milestone: 20.05 → 20.08
James Page (james-page)
Changed in charm-ceph-radosgw:
milestone: 20.08 → none
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Setting the following configuration in ceph.conf should resolve the problem

rgw keystone revocation interval = 0

ceph.conf template [1] should be updated with this configuration

[1] https://opendev.org/openstack/charm-ceph-radosgw/src/branch/master/templates/ceph.conf

Changed in charm-ceph-radosgw:
assignee: nobody → Hemanth Nakkina (hemanth-n)
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

rgw keystone revocation interval config parameter and corresponding thread is removed from ceph octopus

https://github.com/ceph/ceph/commit/6b4c985319224b3fe34f618b3faa07b183b055c8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-radosgw (master)
Changed in charm-ceph-radosgw:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-radosgw (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/792503
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/d9cc3f3bfbb9dbac4095470257a3259fde829461
Submitter: "Zuul (22348)"
Branch: master

commit d9cc3f3bfbb9dbac4095470257a3259fde829461
Author: Hemanth Nakkina <email address hidden>
Date: Fri May 21 11:52:35 2021 +0530

    set rgw keystone revocation interval to 0

    Ceph RGW checks revocation list for every 600 seconds. This is not
    required for non PKI tokens and PKI tokens are removed in OpenStack
    Pike release. This results in unnecessary logs in ceph and keystone.

    Set the rgw keystone revocation interval to 0 in ceph conf. Also
    this parameter is removed in upstream from Ceph Octopus. So ensure
    not to add this parameter from ceph release Octopus.

    Closes-Bug: #1758982
    Change-Id: Iaeb10dc25bb52df9dd3746ecf4fe5859d4efd459

Changed in charm-ceph-radosgw:
status: In Progress → Fix Committed
Changed in charm-ceph-radosgw:
milestone: none → 21.10
Changed in charm-ceph-radosgw:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.