Keystone broken after upgrade from Queens to Rocky

[Lorenzo Cavassa]

Queens cloud 100% working before the upgrade.

Started an action-managed upgrade to Rocky.

Keystone charm release: 292

After the upgrade the Keystone service doesn't work anymore.
Looks like there are issues with the 'python-keystoneclient'.

python3-keystoneclient has been installed.

Here attached the /var/log/apache2/keystone_access.log & /var/log/apache2/keystone_error.log files

[Lorenzo Cavassa (lorenzo-cavassa)]

subscribed ~field-critical

[James Page (james-page)]

2019-03-07 12:30:39 INFO juju-log Configuring Keystone to use a pre-configured admin token.
2019-03-07 12:30:39 INFO juju-log Loaded template from templates/wsgi-openstack-api.conf
2019-03-07 12:30:39 INFO juju-log Rendering from template: /etc/apache2/sites-enabled/wsgi-openstack-api.conf
2019-03-07 12:30:39 INFO juju-log Wrote template /etc/apache2/sites-enabled/wsgi-openstack-api.conf.
2019-03-07 12:30:40 INFO juju-log Loaded template from /var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/templates/memcached.conf
2019-03-07 12:30:40 INFO juju-log Rendering from template: /etc/memcached.conf
2019-03-07 12:30:40 INFO juju-log Wrote template /etc/memcached.conf.
2019-03-07 12:30:40 DEBUG juju-log Database is initialised
2019-03-07 12:30:40 DEBUG config-changed Traceback (most recent call last):
2019-03-07 12:30:40 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 26, in <module>
2019-03-07 12:30:40 DEBUG config-changed from keystoneclient.v2_0 import client
2019-03-07 12:30:40 DEBUG config-changed ModuleNotFoundError: No module named 'keystoneclient'

[James Page (james-page)]

manager launcher is using the openstack-origin to determine which py version to use; in the action managed upgrade case, this won't work so the launcher check needs to look at the installed package version, rather than the openstack-origin configuration option.

[James Page (james-page)]

Also:

unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade Traceback (most recent call last):
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade File "/var/lib/juju/agents/unit-keystone-0/charm/actions/openstack-upgrade", line 59, in <module>
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade openstack_upgrade()
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade File "/var/lib/juju/agents/unit-keystone-0/charm/actions/openstack-upgrade", line 56, in openstack_upgrade
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade os.execl('./hooks/config-changed-postupgrade', '')
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade File "/usr/lib/python3.6/os.py", line 527, in execl
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade execv(file, args)
unit-keystone-0: 13:42:19 DEBUG unit.keystone/0.openstack-upgrade ValueError: execv() arg 2 first element cannot be empty

[James Page (james-page)]

FWIW if you kill the spinning config-changed hook and run the action it mostly dtrt.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/641675

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/641675
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=627ab761dd4375bc1e5d5de3c703136495986d78
Submitter: Zuul
Branch: master

commit 627ab761dd4375bc1e5d5de3c703136495986d78
Author: Alex Kavanagh <email address hidden>
Date: Thu Mar 7 14:11:31 2019 +0000

    Fix manager.py launch python version to match installed payload

    The manager.py helper is launched with either py2 or py3 depending on
    the version of the payload. However, it was using openstack-origin
    rather than actually picking up the actual version of the installed
    software. This patch ensures that the manager.py is run with the
    python version of the installed payload, which resolves a bug when
    upgrading to rocky (which is the py2 to py3 transition of the payload).

    Also fixes an os.execl() bug which changed behaviour from python2 to
    python3.

    Change-Id: I17255b84ca067886f866a5f8311dcab57fba6f14
    Closes-bug: #1818988

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/641730

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/641730
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=0c0b4ed48fc405ad7a7d6640a6bf696c216e5f10
Submitter: Zuul
Branch: stable/18.11

commit 0c0b4ed48fc405ad7a7d6640a6bf696c216e5f10
Author: Alex Kavanagh <email address hidden>
Date: Thu Mar 7 14:11:31 2019 +0000

    Fix manager.py launch python version to match installed payload

    The manager.py helper is launched with either py2 or py3 depending on
    the version of the payload. However, it was using openstack-origin
    rather than actually picking up the actual version of the installed
    software. This patch ensures that the manager.py is run with the
    python version of the installed payload, which resolves a bug when
    upgrading to rocky (which is the py2 to py3 transition of the payload).

    Also fixes an os.execl() bug which changed behaviour from python2 to
    python3.

    Closes-bug: #1818988
    (cherry picked from commit 627ab761dd4375bc1e5d5de3c703136495986d78)

    Change-Id: Ie56374f5882619239f544aaeea4acf92d3d0879d