Keystone checking for PKI directory despite pike being set at openstack-origin

[Michael Skalka]

Keystone charm version 276.

Keystone is failing on a 'cluster-relation-changed" hook in a Pike cloud with the following:

2018-03-27 21:33:12 DEBUG juju-log cluster:22: Running ensure-pki-permissions
2018-03-27 21:33:12 DEBUG cluster-relation-changed Traceback (most recent call last):
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 880, in <module>
2018-03-27 21:33:12 DEBUG cluster-relation-changed main()
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 873, in main
2018-03-27 21:33:12 DEBUG cluster-relation-changed hooks.execute(sys.argv)
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-27 21:33:12 DEBUG cluster-relation-changed self._hooks[hook_name]()
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1449, in wrapped_f
2018-03-27 21:33:12 DEBUG cluster-relation-changed restart_functions)
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-03-27 21:33:12 DEBUG cluster-relation-changed r = lambda_f()
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1448, in <lambda>
2018-03-27 21:33:12 DEBUG cluster-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1650, in _inner_update_certs_if_available
2018-03-27 21:33:12 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 599, in cluster_changed
2018-03-27 21:33:12 DEBUG cluster-relation-changed check_peer_actions()
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1394, in check_peer_actions
2018-03-27 21:33:12 DEBUG cluster-relation-changed ensure_pki_dir_permissions()
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1624, in ensure_pki_dir_permissions
2018-03-27 21:33:12 DEBUG cluster-relation-changed perms=0o775, recurse=True)
2018-03-27 21:33:12 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1337, in ensure_permissions
2018-03-27 21:33:12 DEBUG cluster-relation-changed os.chown(path, uid, gid)
2018-03-27 21:33:12 DEBUG cluster-relation-changed OSError: [Errno 2] No such file or directory: '/var/lib/keystone/juju_ssl/pki'
2018-03-27 21:33:12 ERROR juju.worker.uniter.operation runhook.go:113 hook "cluster-relation-changed" failed: exit status 1

Keystone config:
    charm: cs:xenial/keystone
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: cloud:xenial-pike
      vip: *keystone-vip
      region: *openstack-region
      preferred-api-version: 3
      #admin-password: openstack
      ssl_ca: *ssl_ca
      ssl_cert: *ssl_cert
      ssl_key: *ssl_key
      os-public-hostname: *hostname_keystone_public

[Sean Feole (sfeole)]

Hey Michael, would you be able to supply me with a juju crashdump, https://github.com/juju/juju-crashdump?? I'll take a look at this shortly after. thanks!

[Michael Skalka (mskalka)]

Unmarked private and moved customer log to: https://drive.google.com/open?id=1xT9rGOluwqDDlJoNZpEaTkyZTO6d4z_q

[Michael Skalka (mskalka)]

I can see where the logic in the charm is breaking down here:

check_peer_actions() is being fired before initialize_pki() in the cluster-relation-changed hook. check_peer_actions should be making the same "os_release('keystone-common') >= 'pike' check that initialize_pki before running any PKI-related actions, or ensure_pki_dir_permissions() should make that check.

[Michael Skalka (mskalka)]

Juju-crashdump: https://drive.google.com/open?id=1IG74oQ_D-oEOVfhQ8RKFj7K-kDp1lqKg

[Sean Feole (sfeole)]

Hey Mike, Yea, looks like that we need to add a gate to check CompareOpenStackReleases(os_release('keystone-common')) >= 'pike' in keystone_utils.py under check_peer_actions() possibly some other locations too. Will continue to debug here on my end, thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/557680

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/557680
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=a189c3da807fc4283232dc416b037b095804dcd8
Submitter: Zuul
Branch: master

commit a189c3da807fc4283232dc416b037b095804dcd8
Author: sfeole <email address hidden>
Date: Thu Mar 29 07:34:20 2018 -0400

    Don't ensure pki permissions for releases <= Pike

    Openstack PKI token support was dropped in the Pike release.
    The following update ensures that PKI token validation is
    only run if the release is supported when the sync leader
    broadcasts any service credentials to its peers.

    In this case; if the release is <= pike. then we can sync
    token certs and ensure the pki permissions are valid.
    Otherwise this action will be skipped.

    Closes-Bug: 1759403
    Change-Id: I3d8ba6d3cac3a3505a3722a5082c3a6933a9ef67