config-changed: package install needs to be action-managed-upgrade aware

[Xav Paice]

I deployed a cloud with Xenial, maas 2.3.5, Juju 2.5-beta1. There are 3 neutron-api units, all on LXDs.

I had previously set openstack-origin to the default (distro), and am running through the process of an openstack upgrade to Queens.

I ran:

juju config neutron-api action-managed-upgrade=True

Then, when hooks were completed:
juju config neutron-api openstack-origin=cloud:xenial-queens

This resulted in config-changed hook failure:

2018-11-27 02:15:58 DEBUG config-changed E: Unable to locate package python-neutron-dynamic-routing
2018-11-27 02:15:58 DEBUG config-changed Traceback (most recent call last):
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/config-changed", line 773, in <module>
2018-11-27 02:15:58 DEBUG config-changed main()
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/config-changed", line 766, in main
2018-11-27 02:15:58 DEBUG config-changed hooks.execute(sys.argv)
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/core/hookenv.py", line 909, in execute
2018-11-27 02:15:58 DEBUG config-changed self._hooks[hook_name]()
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1479, in wrapped_f
2018-11-27 02:15:58 DEBUG config-changed stopstart, restart_functions)
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/core/host.py", line 739, in restart_on_change_helper
2018-11-27 02:15:58 DEBUG config-changed r = lambda_f()
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1478, in <lambda>
2018-11-27 02:15:58 DEBUG config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
2018-11-27 02:15:58 DEBUG config-changed return f(*args, **kwargs)
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/config-changed", line 299, in config_changed
2018-11-27 02:15:58 DEBUG config-changed fatal=True)
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/fetch/ubuntu.py", line 228, in apt_install
2018-11-27 02:15:58 DEBUG config-changed _run_apt_command(cmd, fatal)
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/fetch/ubuntu.py", line 591, in _run_apt_command
2018-11-27 02:15:58 DEBUG config-changed retry_message="Couldn't acquire DPKG lock")
2018-11-27 02:15:58 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/fetch/ubuntu.py", line 567, in _run_with_retries
2018-11-27 02:15:58 DEBUG config-changed result = subprocess.check_call(cmd, **kwargs)
2018-11-27 02:15:58 DEBUG config-changed File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2018-11-27 02:15:58 DEBUG config-changed raise CalledProcessError(retcode, cmd)
2018-11-27 02:15:58 DEBUG config-changed subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', '--option=Dpkg::Options::=--force-confold', 'install', 'python-neutron-dynamic-routing']' returned non-zero exit status 100
2018-11-27 02:15:58 ERROR juju.worker.uniter.operation runhook.go:132 hook "config-changed" failed: exit status 1
2018-11-27 02:15:58 INFO juju.worker.uniter resolver.go:115 awaiting error resolution for "config-changed" hook

[James Page (james-page)]

    global CONFIGS
    if not config('action-managed-upgrade'):
        if openstack_upgrade_available('neutron-common'):
            status_set('maintenance', 'Running openstack upgrade')
            do_openstack_upgrade(CONFIGS)

    additional_install_locations(
        config('neutron-plugin'),
        config('openstack-origin')
    )
    status_set('maintenance', 'Installing apt packages')
    apt_install(filter_installed_packages(
                determine_packages(config('openstack-origin'))),
                fatal=True)

[James Page (james-page)]

The config-changed hook does directly install packages; that needs updating to be aware of action managed upgrades.

However if you follow the stepped upgrade approach then you won't actually hit this issue.

Marking Triaged/Low.

[Alvaro Uria (aluria)]

In BootStack, we also run the upgrade procedure via actions.

In my specific case,
1) deployed openstack-on-lxd's bundle-bionic-queens.yaml,
2) enabled action-managed-upgrade (True)
3) updated neutron-api's openstack-origin value from default (distro) to "cloud:bionic-stein"

Per code shared by jamespage, apt_install is run but add_source is not, so packages such as python3-neutron-fwaas can't be located.
"""
2019-08-02 08:41:22 DEBUG config-changed E: Unable to locate package python3-neutron-lbaas
2019-08-02 08:41:22 DEBUG config-changed E: Unable to locate package python3-neutron-fwaas
2019-08-02 08:41:22 DEBUG config-changed E: Unable to locate package python3-neutron-dynamic-routing
2019-08-02 08:41:22 DEBUG config-changed E: Unable to locate package python3-neutron
2019-08-02 08:41:22 DEBUG config-changed E: Unable to locate package python3-networking-hyperv
"""

Workaround is to:
2b) leave action-managed-upgrade disabled (False)
Note: rest of steps are the same

However, I now see a different error (jumping directly from Queens to Stein, as suggested in the official documentation [1]):
"""
root@juju-047355-17:~# apt-get install python3-neutron
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  python-ceilometerclient python-pika-pool python-positional python-retrying
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  python3-neutron
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
206 not fully installed or removed.
Need to get 0 B/1,606 kB of archives.
After this operation, 14.3 MB of additional disk space will be used.
(Reading database ... 54718 files and directories currently installed.)
Preparing to unpack .../python3-neutron_2%3a14.0.2-0ubuntu1~cloud0_all.deb ...
Unpacking python3-neutron (2:14.0.2-0ubuntu1~cloud0) ...
dpkg: error processing archive /var/cache/apt/archives/python3-neutron_2%3a14.0.2-0ubuntu1~cloud0_all.deb (--unpack):
 trying to overwrite '/usr/bin/neutron-api', which is also in package python-neutron 2:12.0.6-0ubuntu3
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/python3-neutron_2%3a14.0.2-0ubuntu1~cloud0_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
"""

1. https://docs.openstack.org/charm-guide/latest/1904.html#upgrading-openstack

[Alvaro Uria (aluria)]

The above error when installing python3-neutron does not happen if the upgrade is stepped:
Note: from distro (bionic queens):
2b) juju config neutron-api --reset action-managed-upgrade
3b) juju config neutron-api openstack-origin='cloud:bionic-rocky'
3c) juju config neutron-api openstack-origin='cloud:bionic-stein'

No errors were found.

The official doc [1] induces to think that a Queens to Stein upgrade is possible without a stop in Rocky.

1. https://docs.openstack.org/charm-guide/latest/1904.html#upgrading-openstack

[Drew Freiberger (afreiberger)]

Subscribing field-medium.

Performing stepped upgrade with action-managed-upgrade documented process fails. neutron-api must not be action-manage-upgraded like other charms until this bug is fixed.

[Drew Freiberger (afreiberger)]

Subscribing field-medium.

Performing stepped upgrade with action-managed-upgrade documented process fails. neutron-api must not be action-manage-upgraded like other charms until this bug is fixed.

[Drew Freiberger (afreiberger)]

As a cowboy to work around this, I temporarily set apt_install fatal=False at:

    global CONFIGS
    if not config('action-managed-upgrade'):
        if openstack_upgrade_available('neutron-common'):
            status_set('maintenance', 'Running openstack upgrade')
            do_openstack_upgrade(CONFIGS)

    additional_install_locations(
        config('neutron-plugin'),
        config('openstack-origin')
    )
    status_set('maintenance', 'Installing apt packages')
    apt_install(filter_installed_packages(
                determine_packages(config('openstack-origin'))),
                fatal=False) # True)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/763196

[Alvaro Uria (aluria)]

The issue got fixed on commit 518ae9a0 [1].

There were two issues involved:
1) As mentioned by James on comments #1 and #2, apt_install was called no matter the value of action-managed-upgrade. apt_install determined which packages (and versions) to install by checking "openstack-origin" (the mentioned commit now installs available updates on the current mirrors).

2) A non-stepped update was tried. Xenial's "distro" packages are Ocata. If an update to Pike, then Queens, would have happened, the upgrade would have worked (however, action-managed-upgrade should have worked and [1] fixes it).

1. https://opendev.org/openstack/charm-neutron-api/commit/518ae9a0

[Alvaro Uria (aluria)]

This test worked as expected:
"""
juju deploy --series xenial neutron-api --config openstack-origin=distro
juju deploy --series xenial keystone --config openstack-origin=distro
juju deploy --series xenial percona-cluster --config source=distro
juju deploy --series xenial rabbitmq-server --config source=distro
juju status
juju add-relation keystone percona-cluster
juju add-relation neutron-api percona-cluster
juju add-relation neutron-api keystone
juju add-relation neutron-api rabbitmq-server
juju config neutron-api action-managed-upgrade=true
juju config neutron-api openstack-origin="cloud:xenial-pike"
juju run-action --wait neutron-api/leader openstack-upgrade
juju config neutron-api openstack-origin="cloud:xenial-queens"
juju run-action --wait neutron-api/leader openstack-upgrade
"""