stein -> train upgrade: upgrade action fails due to missing cinder_policy.d directory

[Paul Goins]

When using action-managed-upgrades=true with openstack-dashboard and upgrading from stein to train, it looks like the upgrades complete, except the exit code of the action indicates a failure.

The stderr of the action contains this traceback:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/actions/openstack-upgrade", line 61, in <module>
    openstack_upgrade()
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/actions/openstack-upgrade", line 57, in openstack_upgrade
    config_changed()
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 781, in wrapped_f
    return f(*args, **kwargs)
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
    return f(*args, **kwargs)
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_hooks.py", line 216, in config_changed
    CONFIGS.write_all()
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 263, in write_all
    return super(HorizonOSConfigRenderer, self).write_all()
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/templating.py", line 334, in write_all
    [self.write(k) for k in six.iterkeys(self.templates)]
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/templating.py", line 334, in <listcomp>
    [self.write(k) for k in six.iterkeys(self.templates)]
  File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/templating.py", line 325, in write
    with open(config_file, 'wb') as out:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/openstack-dashboard/openstack_dashboard/conf/cinder_policy.d/consistencygroup.yaml'

Upon examining the code, it seems that /usr/share/openstack-dashboard/openstack_dashboard/conf/cinder_policy.d/consistencygroup.yaml is something which is relevant up to stein, but not into train.

I'm wondering if the upgrade process removes the /usr/share/openstack-dashboard/openstack_dashboard/conf/cinder_policy.d/ directory and its contents, but that file remained registered within the still-in-memory HorizonOSConfigRenderer, resulting in errors when trying to run the config_changed() code after completing the upgrade.

(Context: the openstack-upgrade action calls do_action_openstack_upgrade, which succeeds, and the above error is actually on the immediately-following call to hooks.horizon_hooks.config_changed().)

The workaround here seems simple: run "juju exec -u <unit> hooks/config-changed" for each unit.

[Alex Kavanagh (ajkavanagh)]

Thanks for the bug report, Paul.

Please could you add the versions of the charms that you are using or the bundle for the deployment? This helps us narrow down where to look for the bug.

Many thanks.

[Paul Goins (vultaire)]

Hi Alex,

This was on 21.04 charms. Specifically, openstack-dashboard is at revision 313.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/815396

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/815396
Committed: https://opendev.org/openstack/charm-openstack-dashboard/commit/52145181c8bc7690d50492ac58eec3164eeb77ff
Submitter: "Zuul (22348)"
Branch: master

commit 52145181c8bc7690d50492ac58eec3164eeb77ff
Author: Billy Olsen <email address hidden>
Date: Mon Oct 25 17:52:06 2021 -0700

    Force resolve configs after openstack-upgrade action

    Force resolve the global configs option after an openstack upgrade
    action is run but before the config_changed hook is run. This ensures
    that the global CONFIGS values are updated to the appropriate templates
    before rendering the update config.

    Change-Id: Id8ed2f6d60cf79e1360ca7587aa771ffbe1af9aa
    Closes-Bug: 1928853