charm install fails because python-apt binary package removed in groovy

[Corey Bryant]

Installation of several reactive openstack charms is failing on groovy due to the removal of the py2 package python-apt. These should be moved to use python3-apt.

2020-10-13 14:44:33 WARNING juju-log Package python-apt has no installation candidate.
2020-10-13 14:45:06 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-aodh-0/charm/reactive/aodh_handlers.py", line 41, in install_packages
    aodh.install()
  File "lib/charm/openstack/aodh.py", line 210, in install
    AodhCharm.singleton.install()
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms_openstack/plugins/classes.py", line 514, in install
    super().install()
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/classes.py", line 605, in install
    super().install()
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 655, in install
    fetch.apt_install(packages, fatal=True)
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 284, in apt_install
    _run_apt_command(cmd, fatal)
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 776, in _run_apt_command
    _run_with_retries(
  File "/var/lib/juju/agents/unit-aodh-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 756, in _run_with_retries
    result = subprocess.check_call(cmd, env=env)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', '--option=Dpkg::Options::=--force-confold', 'install', 'aodh-api', 'aodh-evaluator', 'aodh-expirer', 'aodh-notifier', 'aodh-listener', 'python3-aodh', 'libapache2-mod-wsgi-py3', 'python-apt', 'memcached', 'python3-memcache', 'haproxy']' returned non-zero exit status 100.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757944
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=b9691d07e348a75a73719f19b72a2ec39a4c5030
Submitter: Zuul
Branch: master

commit b9691d07e348a75a73719f19b72a2ec39a4c5030
Author: Corey Bryant <email address hidden>
Date: Tue Oct 13 17:55:08 2020 +0000

    Switch to python3-apt in Victoria

    The python-apt binary package is removed in Ubuntu Groovy.
    Switch the charm to python3-apt.

    Change-Id: Iaf1da59f71898e273ecd33e0186bbd49ea45ce56
    Closes-Bug: #1899680

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757945
Committed: https://git.openstack.org/cgit/openstack/charm-barbican/commit/?id=f6fbcd82a9f34b9f2afd597ec80b5cf6ebaa85cb
Submitter: Zuul
Branch: master

commit f6fbcd82a9f34b9f2afd597ec80b5cf6ebaa85cb
Author: Corey Bryant <email address hidden>
Date: Tue Oct 13 17:50:55 2020 +0000

    Switch to python3-apt in Victoria

    The python-apt binary package is removed in Ubuntu Groovy.
    Switch the charm to python3-apt.

    Change-Id: Ib3da731b3e649bf99d63229c376eb8c8354a72d7
    Closes-Bug: #1899680

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757946
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=83cf6e507b691578dd4e0d077ba5fa3ec2e3c4af
Submitter: Zuul
Branch: master

commit 83cf6e507b691578dd4e0d077ba5fa3ec2e3c4af
Author: Corey Bryant <email address hidden>
Date: Tue Oct 13 17:42:59 2020 +0000

    Switch to python3-apt in Victoria

    The python-apt binary package is removed in Ubuntu Groovy.
    Switch the charm to python3-apt.

    Change-Id: Ibef912c4e7099dfa8d189da717d14e59685a814b
    Closes-Bug: #1899680

[Alex Zero (citadelcore)]

This is NOT fixed. Despite doing a release upgrade from Focal to Groovy, openstack-origin being set to distro, and running the latest version of the charm, it still attempts to install python-apt.

[Alex Kavanagh (ajkavanagh)]

> This is NOT fixed. Despite doing a release upgrade from Focal to Groovy, openstack-origin being set to distro, and running the latest version of the charm, it still attempts to install python-apt.

Hi Alex. Can I just verify that you did a series upgrade of Ubuntu from focal to groovy (20.04LTS to 20.10)? This isn't generally something that is recommended, and it's not a series upgrade that we verify. Is there a reason you needed Groovy (20.10) in particular? It is generally recommended to stay on the LTS versions of Ubuntu and upgrade to the next version of OpenStack (in this case Victoria - I'm guessing?) using the openstack-origin setting.

Any debug logs show the error would be useful too.

Thanks.

[Alex Zero (citadelcore)]

Hey, I've attached the logs for Designate, which among Aodh and Barbican is failing.

I will admit I did upgrade from 20.04 LTS to 20.10, I was not aware at all that this was something that wasn't supported and as a result all the nodes in the cluster are now running on 20.10.

[Alex Zero (citadelcore)]

Thing is this doesn't make any sense, the release selection logic I've read in charmhelpers should not select packages for an older release than the current. What is going on?

[Alex Kavanagh (ajkavanagh)]

> Thing is this doesn't make any sense, the release selection logic I've read in charmhelpers should not select packages for an older release than the current. What is going on?

Sorry, this is a bit confusing. What do you mean "select packages for an older release than the current"?

[Alex Zero (citadelcore)]

Here in the Barbican charm a different set of packages are used on Victoria and up: https://github.com/openstack/charm-barbican/blob/master/src/lib/charm/openstack/barbican.py#L254

However, this does not take effect. As shown in the log I posted these charms will still attempt to install python-apt over python3-apt, which does not make any sense as I am running Victoria, on Groovy.

[Alex Zero (citadelcore)]

I should add, let me know if there are any other logs I could add that would be useful in debugging.

[Alex Kavanagh (ajkavanagh)]

> However, this does not take effect. As shown in the log I posted these charms will still attempt to install python-apt over python3-apt, which does not make any sense as I am running Victoria, on Groovy.

Yes, but they didn't get installed as the series-upgrade doesn't trigger the charm to run particular package upgrades. i.e. series upgrade is designed around the idea that the OpenStack version doesn't change, just the underlying OS. i.e. the upgrade goes from xenial-queens to bionic-queens (distro), and from bionic-ussuri -> focal-ussuri (distro). In order to series upgrade to groovy, it would be necessary to OpenStack upgrade to focal-victoria and then do a series upgrade to groovy-victoria. Upgrading from focal-ussuri to groovy-victoria directly is just something the charms are not designed to handle. But this shouldn't try to install python-apt so something strange *is* going on there. It's possible that the charm is mis-detecting what is it running on and doing the wrong thing.

I don't think, on reviewing them, that our upgrades documentation (https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/upgrade-overview.html) is fully clear on what the charms are capable of handling, as we assume (in the team) that everybody will run on LTS versions.