subprocess.CalledProcessError: returned non-zero exit status 100.

[Felipe Reyes]

Using cs:neutron-api-281 and trying to upgrade bionic-queens to bionic-rocky fails to install apt packages, steps to reproduce:

# set the action-managed-upgrade flat
juju config neutron-api action-managed-upgrade=true openstack-origin=cloud:bionic-rocky

when config-changed hook runs the following exception is raised:

unit-neutron-api-1: 17:45:53 DEBUG unit.neutron-api/1.config-changed Reading package lists...
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed Building dependency tree...
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed Reading state information...
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed E: Unable to locate package python3-neutron-dynamic-routing
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed E: Unable to locate package python3-neutron-lbaas
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed E: Unable to locate package python3-neutron
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed E: Unable to locate package python3-networking-hyperv
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed E: Unable to locate package python3-neutron-fwaas
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed Traceback (most recent call last):
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/config-changed", line 807, in <module>
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed main()
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/config-changed", line 800, in main
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed hooks.execute(sys.argv)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed self._hooks[hook_name]()
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1568, in wrapped_f
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed stopstart, restart_functions)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/core/host.py", line 741, in restart_on_change_helper
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed r = lambda_f()
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1567, in <lambda>
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed return f(*args, **kwargs)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/config-changed", line 288, in config_changed
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed common_upgrade_charm_and_config_changed()
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/config-changed", line 337, in common_upgrade_charm_and_config_changed
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed fatal=True)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/fetch/ubuntu.py", line 268, in apt_install
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed _run_apt_command(cmd, fatal)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/fetch/ubuntu.py", line 762, in _run_apt_command
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed retry_message="Couldn't acquire DPKG lock")
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/var/lib/juju/agents/unit-neutron-api-1/charm/hooks/charmhelpers/fetch/ubuntu.py", line 740, in _run_with_retries
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed result = subprocess.check_call(cmd, env=env)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed raise CalledProcessError(retcode, cmd)
unit-neutron-api-1: 17:45:54 DEBUG unit.neutron-api/1.config-changed subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', '--option=Dpkg::Options::=--force-confold', 'install', 'python3-memcache',
'python3-neutron-dynamic-routing', 'python3-neutron-lbaas', 'python3-neutron', 'python3-networking-hyperv', 'python3-neutron-fwaas']' returned non-zero exit status 100.

This is due to this section of code https://github.com/openstack/charm-neutron-api/blob/stable/19.10/hooks/neutron_api_hooks.py#L325-L337 where initially the do_openstack_upgrade() is skipped since the flag is set, but then config('openstack-origin') is used to determine what packages are missing, where it should be using os_release('neutron-common') to know what's version of openstack currently running.

[Edward Hope-Morley (hopem)]

Patch: https://review.opendev.org/#/c/696762/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696762
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=518ae9a01b7a7e43b411d012fe0ed8e02890d637
Submitter: Zuul
Branch: master

commit 518ae9a01b7a7e43b411d012fe0ed8e02890d637
Author: Felipe Reyes <email address hidden>
Date: Fri Nov 29 18:10:05 2019 -0300

    Use the version of neutron-common to determine the OpenStack version

    There are transient situations where the config option openstack-origin will
    hold the target OpenStack version, so it's not safe to be used to determine
    what packages should be installed in the unit, an accurate method is to use
    the version of the neutron-common package.

    Change-Id: I88693be390f66ba94626e52b949b5573532ea5d7
    Closes-Bug: #1854538