OpenStack Keystone Charm

prevent unnecessary openstack-upgrade on initial install (was: keystone bootstrap fail on latest/edge)

Bug #1989538 reported by JamesLin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Cinder Charm
Fix Released
Critical
Corey Bryant
OpenStack Glance Charm
Fix Released
Critical
Corey Bryant
OpenStack Heat Charm
Fix Released
Critical
Corey Bryant
OpenStack Keystone Charm
Fix Released
Critical
Alex Kavanagh
OpenStack Neutron API Charm
Fix Released
Critical
Corey Bryant
OpenStack Nova Cloud Controller Charm
Fix Released
Critical
Corey Bryant

Bug Description

== keystone ==

Keystone bootstrap failure when deploy this bundle:

```
local_overlay_enabled: False

series: jammy
applications:
  keystone-mysql-router:
    charm: ch:mysql-router
    channel: latest/edge
  vault-mysql-router:
    charm: ch:mysql-router
    channel: latest/edge
  mysql-innodb-cluster:
    channel: latest/edge
    charm: ch:mysql-innodb-cluster
    num_units: 3
  keystone:
    charm: ch:keystone
    num_units: 1
    channel: latest/edge
  vault:
    charm: ch:vault
    num_units: 1
    channel: latest/edge
relations:
  - - vault:certificates
    - mysql-innodb-cluster:certificates
  - - vault:certificates
    - keystone:certificates
  - - keystone:shared-db
    - keystone-mysql-router:shared-db
  - - vault:shared-db
    - vault-mysql-router:shared-db
  - - keystone-mysql-router:db-router
    - mysql-innodb-cluster:db-router
  - - vault-mysql-router:db-router
    - mysql-innodb-cluster:db-router
```

Error msg:

```
unit.keystone/0.config-changed logger.go:60 raise CalledProcessError(retcode, cmd)
2022-09-12 01:54:16 WARNING unit.keystone/0.config-changed logger.go:60 subprocess.CalledProcessError: Command '('keystone-manage', 'bootstrap', '--bootstrap-username', '_charm-keystone-admin', '--bootstrap-password', 'RfGNgwLSwVtBPx6dTZ8LpWHm6F3rL9HSC8Cz38tFKFFL7FJRYfNVzjgfqPXxXjHd', '--bootstrap-project-name', 'admin', '--bootstrap-role-name', 'Admin', '--bootstrap-service-name', 'keystone', '--bootstrap-admin-url', 'http://10.5.1.103:35357/v3', '--bootstrap-public-url', 'http://10.5.1.103:5000/v3', '--bootstrap-internal-url', 'http://10.5.1.103:5000/v3', '--bootstrap-region-id', 'RegionOne')' returned non-zero exit status 1
```

In root:

```
root@juju-39d662-zaza-86e9ebd11c76-0:/home/ubuntu# keystone-manage bootstrap --bootstrap-username _charm-keystone-admin --bootstrap-password RfGNgwLSwVtBPx6dTZ8LpWHm6F3rL9HSC8Cz38tFKFFL7FJRYfNVzjgfqPXxXjHd --bootstrap-project-name admin --bootstrap-role-name Admin --bootstrap-service-name keystone --bootstrap-admin-url http://10.5.1.103:35357/v3 --bootstrap-public-url 'http://10.5.1.103:5000/v3 --bootstrap-internal-url 'http://10.5.1.103:5000/v3 --bootstrap-region-id RegionOne
/etc/keystone/fernet-keys/ does not contain keys, use keystone-manage fernet_setup to create Fernet keys.
```

The `key-setup` file didn't exist at `/etc/keystone/`. So this bug is because the keystone bootstrap fire before `key_setup()`

keystone is working fine at yoga/stable & focal series

== heat, neutron-api, glance, nova-cloud-controller ==

For these charms, deploys on jammy-zed from latest/edge were failing with config-changed hook failures running db sync commands prior to database config being rendered in config files such as heat.conf.

See original description
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :
Download full text (4.4 KiB)

Full traceback is:

unit-keystone-0: 09:50:46 DEBUG juju.worker.uniter.remotestate got leader settings change for keystone/0: ok=true
unit-keystone-0: 09:50:46 INFO unit.keystone/0.juju-log Bootstrapping keystone.
unit-keystone-0: 09:50:49 WARNING unit.keystone/0.config-changed /etc/keystone/fernet-keys/ does not contain keys, use keystone-manage fernet_setup to create Fernet keys.
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed Traceback (most recent call last):
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 937, in <module>
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed main()
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 930, in main
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed hooks.execute(sys.argv)
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/hookenv.py", line 963, in execute
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed self._hooks[hook_name]()
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1900, in wrapped_f
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed return restart_on_change_helper(
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 865, in restart_on_change_helper
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed r = lambda_f()
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1901, in <lambda>
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed (lambda: f(*args, **kwargs)),
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/hardening/harden.py", line 90, in _harden_inner2
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed return f(*args, **kwargs)
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 235, in config_changed
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed do_openstack_upgrade_reexec(configs=CONFIGS)
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 745, in do_openstack_upgrade_reexec
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed do_openstack_upgrade(configs)
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 793, in do_openstack_upgrade
unit-keystone-0: 09:50:50 WARNING unit.keystone/0.config-changed bootstrap_keystone(configs=configs)
unit-keysto...

Read more...

Changed in charm-keystone:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Alex Kavanagh (ajkavanagh)
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The charm, on the first config-changed tries to do an openstack upgrade because it thinks it needs to do an upgrade; instrumenting the `openstack_upgrade_available` function:

unit-keystone-0: 12:00:23 INFO unit.keystone/0.juju-log openstack_upgrade_available:
unit-keystone-0: 12:00:23 INFO unit.keystone/0.juju-log ..src is cloud:jammy-zed
unit-keystone-0: 12:00:24 INFO unit.keystone/0.juju-log ..cur_vers: 2022.1
unit-keystone-0: 12:00:24 INFO unit.keystone/0.juju-log ..avail_vers = 2022.2
unit-keystone-0: 12:00:24 INFO unit.keystone/0.juju-log ..test = True
unit-keystone-0: 12:00:24 INFO unit.keystone/0.juju-log Performing OpenStack upgrade to zed.
unit-keystone-0: 12:00:24 INFO unit.keystone/0.juju-log DEPRECATION WARNING: Function configure_installation_source is being removed on/around 2017-07 : use charmhelpers.fetch.add_source() instead.
unit-keystone-0: 12:00:25 INFO unit.keystone/0.juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold']

e.g. cur_vers is 2022.1 vs 2022.2 as being available. Time to work out what's gone wrong.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

So the reason is breaks is that during the install hook, the /etc/openstack-release file from the openstack-release package is installed from distro (e.g. jammy) and not the jammy-zed cloud-archive.

The initial call to `configure_installation_source` causes the openstack-release package to be installed **before** the cloud archive apt sources are configured (just tracing through the calls), which means it gets the distro version of openstack-release. Then it's cached for the remainder of the install hook. Then, during config-changed, the charm sees the 'wrong' version is installed and tries to do the upgrade which breaks.

The solution is probably either:

1. break or skip the caching in charmhelpers/contrib/openstack/utils.py:get_os_installed_version() for the initial check; maybe using a no_install parameter or similar to get around the issue.
2. Force a re-installation of the openstack-release package after the sources have been configured so that it's at the right version.

Either way, it's a bit broken.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone/+/857729

Changed in charm-keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (master)

Reviewed: https://review.opendev.org/c/openstack/charm-keystone/+/857729
Committed: https://opendev.org/openstack/charm-keystone/commit/2c33c74c8fb78ecfd61060663fb7b1a649fbd7d6
Submitter: "Zuul (22348)"
Branch: master

commit 2c33c74c8fb78ecfd61060663fb7b1a649fbd7d6
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 14 16:27:57 2022 +0100

    Ensure openstack-release package is correct after install hook

    The linked bug shows the install of the charm with openstack-origin set
    to zed. This happens because configure_installation_source() causes the
    openstack-release package to be installed *before* the zed cloud archive
    sources are configured into /etc/apt and an apt update done. This means
    that the openstack-release package says "yoga" despite the zed packages
    actually being installed.

    Then, on the config-changed hook, it sees that the installed version is
    showing as yoga and tries to do an upgrade. This fails, as the charm
    hasn't yet bootstrapped, and the charm tries to bootstrap after
    upgrading the packages.

    There's a few bugs here which are exposed, but the tactical fix is to
    force the openstack-release to match the installed packages.

    Change-Id: I3f47daf6bda6b62ffe4152ede2709f802f0ab606
    Closes-Bug: #1989538

Changed in charm-keystone:
status: In Progress → Fix Committed
Corey Bryant (corey.bryant)
Changed in charm-glance:
status: New → Triaged
Changed in charm-heat:
status: New → Triaged
Changed in charm-neutron-api:
status: New → Triaged
Changed in charm-nova-cloud-controller:
status: New → Triaged
importance: Undecided → Critical
Changed in charm-neutron-api:
importance: Undecided → Critical
Changed in charm-heat:
importance: Undecided → Critical
Changed in charm-glance:
importance: Undecided → Critical
summary: - keystone bootstrap fail on latest/edge
+ prevent unnecessary config-changed hook openstack-upgrade on initial
+ install (was: keystone bootstrap fail on latest/edge)
summary: - prevent unnecessary config-changed hook openstack-upgrade on initial
- install (was: keystone bootstrap fail on latest/edge)
+ prevent unnecessary openstack-upgrade on initial install (was: keystone
+ bootstrap fail on latest/edge)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/860299

Changed in charm-nova-cloud-controller:
status: Triaged → In Progress
Changed in charm-neutron-api:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-api (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-api/+/860300

Changed in charm-glance:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-glance (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/860301

Changed in charm-heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-heat (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-heat/+/860302

Corey Bryant (corey.bryant)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-heat (master)

Reviewed: https://review.opendev.org/c/openstack/charm-heat/+/860302
Committed: https://opendev.org/openstack/charm-heat/commit/f5a61e7c784e48772fdb07aac4782e74b9e09951
Submitter: "Zuul (22348)"
Branch: master

commit f5a61e7c784e48772fdb07aac4782e74b9e09951
Author: Corey Bryant <email address hidden>
Date: Tue Oct 4 15:19:46 2022 +0000

    Ensure openstack-release package is correct after install hook

    The linked bug shows the install of the charm with openstack-origin set
    to zed. This happens because configure_installation_source() causes the
    openstack-release package to be installed *before* the zed cloud archive
    sources are configured into /etc/apt and an apt update done. This means
    that the openstack-release package says "yoga" despite the zed packages
    actually being installed.

    Then, on the config-changed hook, it sees that the installed version is
    showing as yoga and tries to do an upgrade. This fails, as the charm
    hasn't yet bootstrapped, and the charm tries to bootstrap after
    upgrading the packages.

    There's a few bugs here which are exposed, but the tactical fix is to
    force the openstack-release to match the installed packages.

    Closes-Bug: #1989538
    Change-Id: I501aae7f7d818211deaeda945648dfdf63c21d7a

Changed in charm-heat:
status: In Progress → Fix Committed
Changed in charm-neutron-api:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-api (master)

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api/+/860300
Committed: https://opendev.org/openstack/charm-neutron-api/commit/8e708a76b72fa66234f0ef7fd722ae315eae1b12
Submitter: "Zuul (22348)"
Branch: master

commit 8e708a76b72fa66234f0ef7fd722ae315eae1b12
Author: Corey Bryant <email address hidden>
Date: Tue Oct 4 15:24:12 2022 +0000

    Ensure openstack-release package is correct after install hook

    The linked bug shows the install of the charm with openstack-origin set
    to zed. This happens because configure_installation_source() causes the
    openstack-release package to be installed *before* the zed cloud archive
    sources are configured into /etc/apt and an apt update done. This means
    that the openstack-release package says "yoga" despite the zed packages
    actually being installed.

    Then, on the config-changed hook, it sees that the installed version is
    showing as yoga and tries to do an upgrade. This fails, as the charm
    hasn't yet bootstrapped, and the charm tries to bootstrap after
    upgrading the packages.

    There's a few bugs here which are exposed, but the tactical fix is to
    force the openstack-release to match the installed packages.

    Closes-Bug: #1989538
    Change-Id: Icaa2303d2dbbcfc883ea1cd44ce0a77af1ef7250

Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/860299
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/abb8ee551c0abc91c986f2a8482c7462cd7a8e39
Submitter: "Zuul (22348)"
Branch: master

commit abb8ee551c0abc91c986f2a8482c7462cd7a8e39
Author: Corey Bryant <email address hidden>
Date: Tue Oct 4 15:23:55 2022 +0000

    Ensure openstack-release package is correct after install hook

    The linked bug shows the install of the charm with openstack-origin set
    to zed. This happens because configure_installation_source() causes the
    openstack-release package to be installed *before* the zed cloud archive
    sources are configured into /etc/apt and an apt update done. This means
    that the openstack-release package says "yoga" despite the zed packages
    actually being installed.

    Then, on the config-changed hook, it sees that the installed version is
    showing as yoga and tries to do an upgrade. This fails, as the charm
    hasn't yet bootstrapped, and the charm tries to bootstrap after
    upgrading the packages.

    There's a few bugs here which are exposed, but the tactical fix is to
    force the openstack-release to match the installed packages.

    Closes-Bug: #1989538
    Change-Id: Icdef04e25e74c0a18fd49997c5f5b0540d583f40

Felipe Reyes (freyes)
Changed in charm-neutron-api:
assignee: nobody → Corey Bryant (corey.bryant)
Changed in charm-nova-cloud-controller:
assignee: nobody → Corey Bryant (corey.bryant)
Changed in charm-heat:
assignee: nobody → Corey Bryant (corey.bryant)
Corey Bryant (corey.bryant)
Changed in charm-cinder:
status: New → Triaged
importance: Undecided → Critical
Changed in charm-glance:
assignee: nobody → Corey Bryant (corey.bryant)
Changed in charm-cinder:
assignee: nobody → Corey Bryant (corey.bryant)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/860426

Changed in charm-cinder:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-glance (master)

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/860301
Committed: https://opendev.org/openstack/charm-glance/commit/4d29d4792ebc3f99d484488ed1041ede74b4a620
Submitter: "Zuul (22348)"
Branch: master

commit 4d29d4792ebc3f99d484488ed1041ede74b4a620
Author: Corey Bryant <email address hidden>
Date: Tue Oct 4 15:24:26 2022 +0000

    Ensure openstack-release package is correct after install hook

    The linked bug shows the install of the charm with openstack-origin set
    to zed. This happens because configure_installation_source() causes the
    openstack-release package to be installed *before* the zed cloud archive
    sources are configured into /etc/apt and an apt update done. This means
    that the openstack-release package says "yoga" despite the zed packages
    actually being installed.

    Then, on the config-changed hook, it sees that the installed version is
    showing as yoga and tries to do an upgrade. This fails, as the charm
    hasn't yet bootstrapped, and the charm tries to bootstrap after
    upgrading the packages.

    There's a few bugs here which are exposed, but the tactical fix is to
    force the openstack-release to match the installed packages.

    Closes-Bug: #1989538
    Change-Id: Ic1cf491e1121f4fee3e57862fd1cc0315d3953da

Changed in charm-glance:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (master)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/860426
Committed: https://opendev.org/openstack/charm-cinder/commit/0fa3fdadd03eb5ee7178e88f3ad99777d41a3439
Submitter: "Zuul (22348)"
Branch: master

commit 0fa3fdadd03eb5ee7178e88f3ad99777d41a3439
Author: Corey Bryant <email address hidden>
Date: Wed Oct 5 14:42:06 2022 +0000

    Ensure openstack-release package is correct after install hook

    The linked bug shows the install of the charm with openstack-origin set
    to zed. This happens because configure_installation_source() causes the
    openstack-release package to be installed *before* the zed cloud archive
    sources are configured into /etc/apt and an apt update done. This means
    that the openstack-release package says "yoga" despite the zed packages
    actually being installed.

    Then, on the config-changed hook, it sees that the installed version is
    showing as yoga and tries to do an upgrade. This fails, as the charm
    hasn't yet bootstrapped, and the charm tries to bootstrap after
    upgrading the packages.

    There's a few bugs here which are exposed, but the tactical fix is to
    force the openstack-release to match the installed packages.

    Closes-Bug: #1989538
    Change-Id: Ifc35474c98b1ffa1d7d81ac42deb897ace7db885

Changed in charm-cinder:
status: In Progress → Fix Committed
Felipe Reyes (freyes)
Changed in charm-cinder:
status: Fix Committed → Fix Released
Changed in charm-glance:
status: Fix Committed → Fix Released
Changed in charm-heat:
status: Fix Committed → Fix Released
Changed in charm-keystone:
status: Fix Committed → Fix Released
Changed in charm-neutron-api:
status: Fix Committed → Fix Released
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.