openstack-upgrade fails on leader unit during action-managed-upgrade

[Drew Freiberger]

When performing bionic-queens/distro to bionic-rocky upgrade openstack upgrade with action-managed-upgrade=true, the neutron-gateway leader unit failed to execute the writing of the template files. This resulted in a failed action.

2019-11-09 16:04:32 DEBUG openstack-upgrade Traceback (most recent call last):
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/actions/openstack-upgrade", line 47, in <module>
2019-11-09 16:04:32 DEBUG openstack-upgrade openstack_upgrade()
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/actions/openstack-upgrade", line 43, in openstack_upgrade
2019-11-09 16:04:32 DEBUG openstack-upgrade config_changed()
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1568, in wrapped_f
2019-11-09 16:04:32 DEBUG openstack-upgrade stopstart, restart_functions)
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2019-11-09 16:04:32 DEBUG openstack-upgrade r = lambda_f()
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1567, in <lambda>
2019-11-09 16:04:32 DEBUG openstack-upgrade (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
2019-11-09 16:04:32 DEBUG openstack-upgrade return f(*args, **kwargs)
2019-11-09 16:04:32 DEBUG openstack-upgrade File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/neutron_hooks.py", line 154, in config_changed
2019-11-09 16:04:32 DEBUG openstack-upgrade CONFIGS.write_all()
2019-11-09 16:04:32 DEBUG openstack-upgrade AttributeError: 'NoneType' object has no attribute 'write_all'
2019-11-09 16:04:33 DEBUG juju.worker.uniter.operation executor.go:90 committing operation "run action b6c0bf8d-96e2-4f29-8df0-4d1f2e15691f"

$ juju show-action-output b6c0bf8d-96e2-4f29-8df0-4d1f2e15691f
message: exit status 1
results:
  outcome: success, upgrade completed.
status: failed
timing:
  completed: 2019-11-09 16:04:33 +0000 UTC
  enqueued: 2019-11-09 15:58:43 +0000 UTC
  started: 2019-11-09 15:58:45 +0000 UTC

It appears there may be an issue with the wrapper around the action not setting the proper CONFIGS context. No apparent negative side-effects. Agents running, charm/unit status in good health, packages upgraded.

[Drew Freiberger (afreiberger)]

Subscribed field-medium

Issue is present in 19.10 charms when upgrading from bionic-queens to rocky.

[Drew Freiberger (afreiberger)]

This appears to possibly be affecting several charms from the openstack-upgrade layer.

I see it hints of it in openstack-dashboard as well:

DDMI-OK-NONPROD jujumanage@CMOOSINCFUP7301:~/upgrades$ juju run-action --wait openstack-dashboard/3 openstack-upgrade
unit-openstack-dashboard-3:
  id: 7bbfb5ea-94ee-4d61-893c-1b5cda2a4efd
  message: do_openstack_upgrade resulted in an unexpected error
  results:
    outcome: upgrade failed, see traceback.
    traceback: |
      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-openstack-dashboard-3/charm/charmhelpers/contrib/openstack/utils.py", line 1244, in do_action_openstack_upgrade
          upgrade_callback(configs=configs)
        File "/var/lib/juju/agents/unit-openstack-dashboard-3/charm/hooks/horizon_utils.py", line 349, in do_openstack_upgrade
          configs.set_release(openstack_release=new_os_rel)
      AttributeError: 'NoneType' object has no attribute 'set_release'
  status: failed
  timing:
    completed: 2019-11-09 20:54:57 +0000 UTC
    enqueued: 2019-11-09 20:52:00 +0000 UTC
    started: 2019-11-09 20:52:00 +0000 UTC
  unit: openstack-dashboard/3

[Drew Freiberger (afreiberger)]

I will note that I ran through openstack upgrade process pretty cleanly and didn't experience these issues on 19.07 charms, it seems a change in 19.10.

[Drew Freiberger (afreiberger)]

Here's another thread on this issue:

https://discourse.jujucharms.com/t/error-with-openstack-dasboard-upgrade/2442

This is the function failing:

https://github.com/openstack/charm-openstack-dashboard/blob/stable/19.10/hooks/horizon_utils.py#L323-L349

configs is passed from the action code:

https://github.com/openstack/charm-openstack-dashboard/blob/stable/19.10/actions/openstack_upgrade.py#L41-L55

But CONFIGS is not initialized as is required by the new structure of the CONFIGS setup in horizon_hooks.py:

https://github.com/openstack/charm-openstack-dashboard/blob/stable/19.10/hooks/horizon_hooks.py#L108-L110

It appears you're to import CONFIGS and then execute the resolve_CONFIGS function to populate the configs.

I think the fix is going to be additionally importing the resolve_CONFIGS from horizon_hooks, and running that to setup configs before calling do_openstack_upgrade within the actions/openstack_upgrade.py code.

[Drew Freiberger (afreiberger)]

Upgrading to field-high. Confirmed bug, notes on the issue and suggested fix in comment #4.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699720
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=f7ef6a2af6adabd9e00ea6e09bc95a1d618e13ab
Submitter: Zuul
Branch: master

commit f7ef6a2af6adabd9e00ea6e09bc95a1d618e13ab
Author: Drew Freiberger <email address hidden>
Date: Wed Dec 18 09:57:03 2019 -0600

    Add resolve_CONFIGS to action-managed-upgrade

    In change-id I08a847abe7db22a11860f359442e1e8945585466 which
    addresses bug #1844325, CONFIGS must be loaded by any function
    needing to import and use CONFIGS. When running openstack-upgrade
    via juju actions, CONFIGS = None causing traceback when running
    configs.set_release(new_release). This change provides the
    CONFIGS resolution to enable action managed upgrades.

    Change-Id: I9258d798f9d6d75822cd5446ee039308a43da867
    Closes-Bug: #1851943