keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request.

[Andrew McLeod]

When deploying ceph-radosgw charm for pike (proposed) on s390x, I get the following status error:

ceph-radosgw/0* error idle 16 10.0.8.120 80/tcp hook failed: "identity-service-relation-changed" for keystone:identity-service

The ceph-radosgw logs indicate this is caused by:

http://pastebin.ubuntu.com/25265602/

Looking at the keystone unit, the error there appears to be the following:

(keystone.common.wsgi): 2017-08-07 21:58:42,756 ERROR [Errno 2] No such file or directory: '/var/lib/keystone/juju_ssl/pki/certs/ca.pem'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 228, in __call__
    result = method(req, **params)
  File "/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 178, in wrapped
    return func_or_cls(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 74, in ca_cert
    with open(CONF.signing.ca_certs, 'r') as ca_file:
IOError: [Errno 2] No such file or directory: '/var/lib/keystone/juju_ssl/pki/certs/ca.pem'

Charm deploy config:

  ceph-radosgw:
    annotations:
      gui-x: '1000'
      gui-y: '250'
    charm: cs:~openstack-charmers-next/ceph-radosgw
    num_units: 1
    options:
      source: cloud:pike/proposed

I believe this may also affect arm64.

[James Page (james-page)]

PKI support dropped in Pike, need to update keystone not to try to throw that data across the link.

[James Page (james-page)]

OK so the challenge here is that the radosgw charm requests the signing cert + ca from keystone to deal with revocation of PKI formatted tokens; these are dropped as of Pike, so that information is never provided from keystone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492553

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492554

[James Page (james-page)]

To confirm that revocation lists no longer have context without PKI:

hi
15:29 (hopefully) quick question about token revocation lists - do/did they only apply for PKI tokens? or do they also apply for UUID and Fernet formats as well?
15:30 we're dropping PKI support in the keystone charms they cycle - just figuring out what we do with regards to certs and ca files related to signing of revocation requests..
15:32 → ducttap__ and PsionTheory joined ⇐ tobberydberg, spzala and sbezverk quit ↔ tobberyd_ and ducttape_ popped in ↔ sjain nipped out
15:58 K<kmalloc> Morgan Fainberg jamespage: recommend not using them at all
15:59 J<jamespage> James Page kmalloc: that was what I thought
15:59 thanks for confirming
15:59 J— jamespage does not have to throw away the last hours work now :-)
15:59 → @lbragstad (opped) joined
15:59 K<kmalloc> Morgan Fainberg jamespage: use fernet tokens, do not use the revocation list (even with uuid tokens). Largely it was for pki tokens, but easiest bet turn off "revoke by id" and ignore the rev list
15:59 :)
16:00 jamespage: happy to make your life easier
16:00 J<jamespage> James Page :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492553
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=ef3529161a48b5c34105cab007cefed136ecd1ec
Submitter: Jenkins
Branch: master

commit ef3529161a48b5c34105cab007cefed136ecd1ec
Author: James Page <email address hidden>
Date: Thu Aug 10 15:39:02 2017 +0100

    keystone: PKI token format removal

    As of Pike, the OpenStack charms no longer generate the certificates
    and CA used to sign token revocation lists as this is associated with
    the PKI token format, which has been removed from OpenStack in
    favor of UUID or Fernet formats.

    Soft-fail on cert retrieval if an InternalServerError is thrown; this
    is most likely due to the fact that the keystone WSGI server cannot
    find the relevant files on the underlying filesystem.

    Change-Id: Ib592e7e47e10bed2d59c9136a3267f9c7ce8da83
    Closes-Bug: 1709189

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492554
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=ee45612e7c3dd78712a0ad29576de557d144a1fd
Submitter: Zuul
Branch: master

commit ee45612e7c3dd78712a0ad29576de557d144a1fd
Author: James Page <email address hidden>
Date: Thu Aug 10 15:22:50 2017 +0100

    pki: conditional enablement of signing section

    Only enable the [signing] section of the keystone configuration
    if PKI token format is in use; other token formats don't have
    support for token revocation retrieval.

    Note that PKI format tokens are no longer supported >= Pike.

    Change-Id: I8179ecc5d37d866588147f639ebc77a870408dfe
    Closes-Bug: 1709189