[Charms 21.04] start hook fails after upgrade to Wallaby

[Radu Malica]

I am running openstack version Victoria on Focal Ubuntu deployed with Juju charms and i wanted to upgrade to Wallaby

Everything worked fine by the book until i ran the upgrade on Masakari app with the following commands:

juju config masakari action-managed-upgrade=True
juju config masakari openstack-origin=cloud:focal-wallaby

charm version is 11.0.0, revision 11 (latest):
masakari 11.0.0 error 3 masakari charmstore 11 ubuntu hook failed: "start"

The Masakari stack is having 3 instances with HA cluster deployed. The HA cluster upgraded successfully

masakari/6 error idle 4/lxd/4 10.118.0.138 15868/tcp hook failed: "start"
  masakari-hacluster/3 active idle 10.118.0.138 Unit is ready and clustered
masakari/7 error idle 5/lxd/1 10.118.0.137 15868/tcp hook failed: "start"
  masakari-hacluster/5 active idle 10.118.0.137 Unit is ready and clustered
masakari/8* error idle 3/lxd/6 10.118.0.136 15868/tcp hook failed: "start"
  masakari-hacluster/4* active idle 10.118.0.136 Unit is ready and clustered

From the logs on each instances i can see this error:

unit-masakari-7: 07:24:57 ERROR unit.masakari/7.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-masakari-7/charm/reactive/masakari_handlers.py", line 40, in render_config
    charm_class.upgrade_if_available(args)
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1142, in upgrade_if_available
    if self.openstack_upgrade_available(self.release_pkg):
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1109, in openstack_upgrade_available
    raise RuntimeError(
RuntimeError: In charms_openstack.charm.core.openstack_upgrade_available() cur_vers=None and avail_vers=2021.1, one of which is None. This usually implies that the openstack version is not present in the self.package_codenames or os_utils.PACKAGE_CODENAMES. Please re-visit and fix.

unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start Traceback (most recent call last):
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/charm/hooks/start", line 22, in <module>
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start main()
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start bus.dispatch(restricted=restricted_mode)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start _invoke(other_handlers)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start handler.invoke()
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start self._action(*args)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/charm/reactive/masakari_handlers.py", line 40, in render_config
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start charm_class.upgrade_if_available(args)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1142, in upgrade_if_available
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start if self.openstack_upgrade_available(self.release_pkg):
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1109, in openstack_upgrade_available
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start raise RuntimeError(
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start RuntimeError: In charms_openstack.charm.core.openstack_upgrade_available() cur_vers=None and avail_vers=2021.1, one of which is None. This usually implies that the openstack version is not present in the self.package_codenames or os_utils.PACKAGE_CODENAMES. Please re-visit and fix.
unit-masakari-7: 07:24:58 ERROR juju.worker.uniter.operation hook "start" (via explicit, bespoke hook script) failed: exit status 1

It seems the charm is unable to get the openstack release from the OS, which is the same LXD containers with Focal as all other openstack components.

From the error above i am assuming that Masakari didn't even upgrade to Wallaby, since the script cannot find the cur_vers of the Openstack (which was Victoria and running properly)

Thanks

[Frode Nordahl (fnordahl)]

This is happening due to https://review.opendev.org/c/openstack/charms.openstack/+/787641 missing in the stable/21.04 charms.

[Radu Malica (radumalica)]

Thanks for the clarification, is there any workaround for this , or should I wait for the openstack-charmers to update the masakari charm ?

[Frode Nordahl (fnordahl)]

This would be an issue for all the stable reactive charms at this point in time, the only available workaround would be to switch over to the -next charms that track the main branch (cs:~openstack-charmers-next/CHARM). There is of course risk involved in doing that as we are still in the development cycle and they have not received pre-release testing. So if this is for a production environment it would most likely be best to wait until the issue is resolved for all the reactive charms.

[Aurelien Lourot (aurelien-lourot)]

https://review.opendev.org/c/openstack/charm-masakari/+/805810 has now landed, which must have triggered a rebuild and picked up https://review.opendev.org/c/openstack/charms.openstack/+/787641 , which solves this bug.

Now creating a 21.04 batch for all reactive charms: https://review.opendev.org/q/topic:wallaby-bundles-stable%252F21.04

[Chris MacNaughton (chris.macnaughton)]

The 21.20 release has now gone out so I'm marking the rest of the in-progress targets as fix-released!

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Page <email address hidden>" on branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-octavia-dashboard/+/806227
Reason: Review targets obsolete stable branch