[rocky][19.04] Upgrading a deployment from Queens to Rocky resulted in purging of neutron-l3-agent package

[Dmitrii Shcherbakov]

Upgrading OpenStack from Queens to Rocky results in removal of neutron-l3-agent package which leads to a networking outage.

The issue with the fact that first python3 packages are installed and then python2 packages are purged. The neutron-l3-agent package depends on both.

At the same time neutron-dhcp-agent survives the upgrade while both packages seem to have identical "Depends" directives for python[3]-neutron:

https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/neutron/tree/debian/control?h=stable/rocky#n178
Package: neutron-l3-agent
Architecture: all
Depends:
 conntrack,
 iputils-arping,
 keepalived,
 neutron-metadata-agent (= ${source:Version}),
 python-neutron (= ${source:Version}) | python3-neutron (= ${source:Version}),
 python-neutron-fwaas (>= 1:13.0.0~) | python3-neutron-fwaas (>= 1:13.0.0~),

https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/neutron/tree/debian/control?h=stable/rocky#n156
Package: neutron-dhcp-agent
Architecture: all
Depends:
 dnsmasq-base (>= 2.76),
 dnsmasq-utils (>= 2.76),
 neutron-metadata-agent (= ${source:Version}),
 python-neutron (= ${source:Version}) | python3-neutron (= ${source:Version}),

When I try to install neutron-l3-agent it tries to pull python2 dependencies again (even though the python3 ones are already there):
https://paste.ubuntu.com/p/J7Vq8KKnBf/

More info below:

------
2019-05-07 16:42:41 DEBUG neutron-plugin-api-relation-changed Selecting previously unselected package neutron-l3-agent.
2019-05-07 16:42:41 DEBUG neutron-plugin-api-relation-changed Preparing to unpack .../6-neutron-l3-agent_2%3a12.0.5-0ubuntu1_all.deb ...
2019-05-07 16:42:41 DEBUG neutron-plugin-api-relation-changed Unpacking neutron-l3-agent (2:12.0.5-0ubuntu1) ...

2019-05-07 16:42:47 DEBUG neutron-plugin-api-relation-changed Setting up neutron-l3-agent (2:12.0.5-0ubuntu1) ...
2019-05-07 16:42:47 DEBUG neutron-plugin-api-relation-changed Created symlink /etc/systemd/system/multi-user.target.wants/neutron-l3-agent.service → /lib/systemd/syst
em/neutron-l3-agent.service.

/var/log/apt/term.log:

update-alternatives: using /usr/bin/python3-nova-metadata-wsgi to provide /usr/bin/nova-metadata-wsgi (nova-metadata-wsgi) in auto mode
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 113229 files and directories currently installed.)
Purging configuration files for python-neutron (2:13.0.2-0ubuntu1~cloud0) ...
Purging configuration files for neutron-l3-agent (2:13.0.2-0ubuntu1~cloud0) ...
Purging configuration files for python-neutron-fwaas (1:13.0.1-0ubuntu1~cloud0) ...
Purging configuration files for python-nova (2:18.1.0-0ubuntu1~cloud0) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.21) ...
Log ended: 2019-05-08 13:46:55

/var/log/apt/history.log:

# install python3 packages
Start-Date: 2019-05-08 13:45:18
Commandline: apt-get --assume-yes --option=Dpkg::Options::=--force-confold install nova-compute genisoimage librbd1 xfsprogs nfs-common ceph-common nova-compute-kvm n
ova-api-metadata python3-nova python3-memcache python3-rados python3-rbd python3-neutron

# purge python2 packages
Start-Date: 2019-05-08 13:46:40
Commandline: apt-get --assume-yes purge python-neutron python-nova
Purge: neutron-l3-agent:amd64 (2:13.0.2-0ubuntu1~cloud0), python-nova:amd64 (2:18.1.0-0ubuntu1~cloud0), python-neutron:amd64 (2:13.0.2-0ubuntu1~cloud0), python-neutron-fwaas:amd64 (1:13.0.1-0ubuntu1~cloud0)
End-Date: 2019-05-08 13:46:55

apt policy python3-neutron
https://paste.ubuntu.com/p/3vKsFBf9Hn/

apt-rdepends:

neutron-l3-agent
  Depends: conntrack
  Depends: iputils-arping
  Depends: keepalived
  Depends: neutron-metadata-agent (= 2:13.0.2-0ubuntu1~cloud0)
  Depends: python-neutron (= 2:13.0.2-0ubuntu1~cloud0)
  Depends: python-neutron-fwaas (>= 1:13.0.0~)
  Depends: python3-neutron (= 2:13.0.2-0ubuntu1~cloud0)
  Depends: python3-neutron-fwaas (>= 1:13.0.0~)
# ...

neutron-dhcp-agent
  Depends: dnsmasq-base (>= 2.76)
  Depends: dnsmasq-utils (>= 2.76)
  Depends: neutron-metadata-agent (= 2:13.0.2-0ubuntu1~cloud0)
  Depends: python-neutron (= 2:13.0.2-0ubuntu1~cloud0)
  Depends: python3-neutron (= 2:13.0.2-0ubuntu1~cloud0)
# ...

Charm code:
https://opendev.org/openstack/charm-neutron-openvswitch/src/branch/stable/19.04/hooks/neutron_ovs_utils.py#L323-L325

    if cmp_release >= 'rocky':
        return PURGE_PACKAGES
    return []

https://opendev.org/openstack/charm-neutron-openvswitch/src/branch/master/hooks/neutron_ovs_utils.py#L119-L122
PURGE_PACKAGES = [
    'python-neutron',
    'python-neutron-fwaas',
]

https://opendev.org/openstack/charm-neutron-openvswitch/src/branch/master/hooks/neutron_ovs_hooks.py#L115-L124

    install_packages()
    install_tmpfilesd()

    # NOTE(jamespage): purge any packages as a result of py3 switch
    # at rocky.
    packages_to_purge = determine_purge_packages()
    request_nova_compute_restart = False
    if packages_to_purge:
        purge_packages(packages_to_purge)
        request_nova_compute_restart = True

[Dmitrii Shcherbakov (dmitriis)]

Subscribed ~field-critical.

[Dmitrii Shcherbakov (dmitriis)]

Priority usage of python2 dependencies is reproducible on a clean environment:

lxc launch ubuntu:bionic neutrontest
Creating neutrontest
Starting neutrontest

lxc exec neutrontest bash

root@neutrontest:~# sudo add-apt-repository cloud-archive:rocky
 Ubuntu Cloud Archive for OpenStack Rocky
 More info: https://wiki.ubuntu.com/ServerTeam/CloudArchive
Press [ENTER] to continue or Ctrl-c to cancel adding it.

Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  ubuntu-cloud-keyring
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 6008 B of archives.
After this operation, 22.5 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ubuntu-cloud-keyring all 2018.09.18.1~18.04.0 [6008 B]
Fetched 6008 B in 1s (8364 B/s)
Selecting previously unselected package ubuntu-cloud-keyring.
(Reading database ... 28640 files and directories currently installed.)
Preparing to unpack .../ubuntu-cloud-keyring_2018.09.18.1~18.04.0_all.deb ...
Unpacking ubuntu-cloud-keyring (2018.09.18.1~18.04.0) ...
Setting up ubuntu-cloud-keyring (2018.09.18.1~18.04.0) ...
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [8570 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [332 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/universe Translation-en [4941 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [119 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [242 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [139 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [4008 B]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2060 B]
Get:13 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [151 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/multiverse Translation-en [108 kB]
Get...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Corey Bryant (corey.bryant)]

I wasn't able to recreate this with packages only, please see attached. Next I'll try recreating with the charms and see if that gives any hints.

[Corey Bryant (corey.bryant)]

I also wasn't able to recreate this with a charm upgrade from queens->rocky, please see attached. I wonder if this is triggered only with a specific config option.

[Corey Bryant (corey.bryant)]

As for priority of py2 packages in rocky. In Rocky, the way things work is that py2 packages are installed by default. If both the py2 and py3 packages are installed, then py2 packages get priority. And if only py3 packages are installed, then they will be used as the dependency.

[Corey Bryant (corey.bryant)]

"When I try to install neutron-l3-agent it tries to pull python2 dependencies again (even though the python3 ones are already there):
https://paste.ubuntu.com/p/J7Vq8KKnBf/"

I wasn't able to recreate what's stated either, please see attached. Is it possible that you were missing python3-neutron-fwaas?

[Corey Bryant (corey.bryant)]

@Dmitrii, Please can you provide some more details to help me recreate this? Did you hit this with the neutron-gateway charm? If so, if you can provide a bundle that would be great. In the mean time I'm just going to move this to Incomplete until we get more details to help with recreation.

[Corey Bryant (corey.bryant)]

Some details on the deployment scenario from Dmitrii:

Dmitrii-Sh> coreycb: https://pastebin.canonical.com/p/HM24kpkbhJ/ (you can cut out the irrelevant parts like saml and ceph)

Dmitrii-Sh> coreycb: that's with https://review.opendev.org/#/c/624495
4:16 PM so, technically, every node is neutron-gateway
4:16 PM and every node runs the neutron-l3-agent
4:16 PM it would be the same with neutron-gateway and DVR
4:16 PM so, consider it a DVR deployment

[Corey Bryant (corey.bryant)]

I'm using the following snippet of config and wasn't yet able to recreate. I'm probably missing something in the config.

Looking at the neutron-ovs charm code it seems like this bug might recreate if use_dvr() evaluates to false (ie. would prevent python3-neutron-fwaas from being installed). I think we will always need to install python3-neutron-fwaas when neutron-l3-agent is installed in order to enable the upgrade of neutron-l3-agent to py3.

@Dmitrii, I'll try this scenario ^. If you think that's the scenario you were hitting, let me know. I don't think it matches up with the bundle config you shared though so we'll see.

[Dmitrii Shcherbakov (dmitriis)]

`enable-dvr: true` is set in my bundle on neutron-api and this is propagated to neutron-openvswitch to make a decision about installing neutron-l3-agent.

[Dmitrii Shcherbakov (dmitriis)]

In other words use_dvr will evaluate to "True" in my case and python3-neutron-fwaas will be installed:

    if use_dvr():
        pkgs.extend(DVR_PACKAGES)
        py3_pkgs.append('python3-neutron-fwaas')

I'm doing another Queens -> Rocky upgrade from scratch and I will provide an archive with /var/log/*

[Dmitrii Shcherbakov (dmitriis)]

Reproduced it.

/etc/ and /var/log/ from a neutron-openvswitch unit:

https://private-fileshare.canonical.com/~dima/charm-dumps/09-05-2019-openvswitch-4-var-log-etc.tar.gz

[Dmitrii Shcherbakov (dmitriis)]

Corey,

Relation data (enable-dvr: "True"): https://paste.ubuntu.com/p/zPs3kB8t4d/

ubuntu@adze:~$ sudo grep -RiP python3-neutron-fwaas /var/log/juju/ ; echo $?
1
ubuntu@adze:~$ grep -RiP python3-neutron-fwaas /var/log/apt/ ; echo $?
1

What can be seen below is that python3-neutron-fwaas is not installed but neutron-l3-agent depends on it while the python2 version is gone.

So you are right about the lack of python3-neutron-fwaas.

-----

term.log:

update-alternatives: using /usr/bin/python3-neutron-l3-agent to provide /usr/bin/neutron-l3-agent (neutron-l3-agent) in auto mode

Purging configuration files for python-neutron (2:13.0.2-0ubuntu3.1~cloud0) ...
Purging configuration files for neutron-l3-agent (2:13.0.2-0ubuntu3.1~cloud0) ...
Purging configuration files for python-neutron-fwaas (1:13.0.1-0ubuntu1~cloud0) ...

history.log:

Start-Date: 2019-05-08 22:52:31
Commandline: apt-get --assume-yes purge python-neutron python-nova
Purge: neutron-l3-agent:amd64 (2:13.0.2-0ubuntu3.1~cloud0), python-nova:amd64 (2:18.1.0-0ubuntu1~cloud0), python-neutron:amd64 (2:13.0.2-0ubuntu3.1~cloud0), python-neutron-fwaas:amd64 (1:13.0.1-0ubuntu1~cloud0)
End-Date: 2019-05-08 22:52:40

ubuntu@adze:~$ apt policy python3-neutron
python3-neutron:
  Installed: 2:13.0.2-0ubuntu3.1~cloud0
  Candidate: 2:13.0.2-0ubuntu3.1~cloud0
  Version table:
 *** 2:13.0.2-0ubuntu3.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
ubuntu@adze:~$ apt policy neutron-l3-agent

neutron-l3-agent:
  Installed: (none)
  Candidate: 2:13.0.2-0ubuntu3.1~cloud0
  Version table:
     2:13.0.2-0ubuntu3.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
     2:12.0.5-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

ubuntu@adze:~$ apt-cache depends neutron-l3-agent
neutron-l3-agent
  Depends: conntrack
  Depends: iputils-arping
  Depends: keepalived
  Depends: neutron-metadata-agent
 |Depends: python-neutron
  Depends: python3-neutron
 |Depends: python-neutron-fwaas
  Depends: python3-neutron-fwaas
  Depends: radvd

ubuntu@adze:~$ apt policy python3-neutron-fwaas
python3-neutron-fwaas:
  Installed: (none)
  Candidate: 1:13.0.1-0ubuntu1~cloud0
  Version table:
     1:13.0.1-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages

ubuntu@adze:~$ apt policy python-neutron-fwaas
python-neutron-fwaas:
  Installed: (none)
  Candidate: 1:13.0.1-0ubuntu1~cloud0
  Version table:
     1:13.0.1-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
     1:12.0.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     1:12.0.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
     1:10.1.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-proposed/ocata/main...

[Dmitrii Shcherbakov (dmitriis)]

Based on what I see, as a subordinate, charm-neutron-openvswitch does not get notified about the upgrade of the nova-compute charm.

python-neutron-fwaas gets purged as a result of the nova-compute charm upgrade which touches some neutron packages. And neutron-openvswitch does not run any package installation steps as it does not receive a config-changed event.

I did it manually and the necessary packages got installed:

juju run --unit neutron-openvswitch/4 'hooks/config-changed'

unit-neutron-openvswitch-4: 23:16:28 TRACE juju.worker.uniter.resolver running op: run action 43b7a6fe-3399-488f-86a0-8499d2ab891b
unit-neutron-openvswitch-4: 23:16:30 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/neutron.conf
unit-neutron-openvswitch-4: 23:16:30 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/plugins/ml2/openvswitch_agent.ini
unit-neutron-openvswitch-4: 23:16:30 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/l3_agent.ini
unit-neutron-openvswitch-4: 23:16:30 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/fwaas_driver.ini
unit-neutron-openvswitch-4: 23:16:31 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/metadata_agent.ini
unit-neutron-openvswitch-4: 23:16:31 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/dhcp_agent.ini
unit-neutron-openvswitch-4: 23:16:31 INFO unit.neutron-openvswitch/4.juju-log Registered config file: /etc/neutron/dnsmasq.conf
unit-neutron-openvswitch-4: 23:16:40 INFO unit.neutron-openvswitch/4.juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
unit-neutron-openvswitch-4: 23:16:44 INFO unit.neutron-openvswitch/4.juju-log Installing ['neutron-l3-agent', 'python3-neutron-fwaas'] with options: ['--option=Dpkg::Options::=--force-confold']
unit-neutron-openvswitch-4: 23:16:57 DEBUG unit.neutron-openvswitch/4.juju-log Updating sysctl_file: /etc/sysctl.d/50-openvswitch.conf values: {'net.ipv4.neigh.default.gc_thresh1': 128, 'net.ipv4.neigh.default.gc_thresh2': 28672, 'net.ipv4.neigh.default.gc_thresh3': 32768, 'net.ipv6.neigh.default.gc_thresh1': 128, 'net.ipv6.neigh.default.gc_thresh2': 28672, 'net.ipv6.neigh.default.gc_thresh3': 32768, 'net.nf_conntrack_max': 1000000, 'net.netfilter.nf_conntrack_buckets': 204800, 'net.netfilter.nf_conntrack_max': 1000000}
unit-neutron-openvswitch-4: 23:16:58 INFO unit.neutron-openvswitch/4.juju-log Creating bridge br-int
unit-neutron-openvswitch-4: 23:16:58 INFO unit.neutron-openvswitch/4.juju-log Creating bridge br-ex
unit-neutron-openvswitch-4: 23:16:59 INFO unit.neutron-openvswitch/4.juju-log Creating bridge br-data
unit-neutron-openvswitch-4: 23:16:59 DEBUG unit.neutron-openvswitch/4.juju-log Interface enp4s0f1 is not a Linux bridge
unit-neutron-openvswitch-4: 23:16:59 INFO unit.neutron-openvswitch/4.juju-log Adding port enp4s0f1 to bridge br-data
unit-neutron-openvswitch-4: 23:17:00 WARNING unit.neutron-openvswitch/4.juju-log prevent-arp-spoofing is True yet this feature is deprecated and no longer has any effect in your version of Openstack
unit-neutron-openvswitch-4: 23:17:00 DEBUG...

[Dmitrii Shcherbakov (dmitriis)]

Adding neutron-openvswitch and nova-compute charms because:

1) nova-compute charm needs to make sure neutron-l3-agent is not accidentally purged before python3-packages are installed by neutron-openvswitch;

2) neutron-openvswitch needs to be notified about the primary charm upgrade.

If nova-compute is not modified and only neutron-openvswitch is then there will be a period of time after purge and before config-changed when Neutron networking will be completely gone as a result of removal of neutron-l3-agent.

[Corey Bryant (corey.bryant)]

Here's how we currently handle subordinate package dependencies in charm-nova-compute. See commit 32ef5b4c and 280c0d7d. For this bug I think we can fix it in this code with our current approach but it's clearly a bit awkward to handle subordinate dependencies in nova-compute. I'd like to also open another bug to see if we can figure out a better way to tell subordinates they need to upgrade and allow them to upgrade their own packages.

def determine_packages()
    ...
    if cmp_release >= 'rocky':
        packages = [p for p in packages if not p.startswith('python-')]
        packages.extend(PY3_PACKAGES)
        if filter_missing_packages(['python-ceilometer']):
            packages.append('python3-ceilometer')
        if filter_missing_packages(['python-neutron']):
            packages.append('python3-neutron')
        if filter_missing_packages(['python-neutron-fwaas']): # to be added for this bug
            packages.append('python3-neutron-fwaas') # to be added for this bug
        if virt_type == 'lxd':
            packages.append('python3-nova-lxd')

    return packages

[Corey Bryant (corey.bryant)]

I think this is limited to nova-compute. I'll target the new bug at neutron-openvswitch.

[Corey Bryant (corey.bryant)]

https://review.opendev.org/#/c/658062/

[Corey Bryant (corey.bryant)]

Here's the other bug that I opened to figure out a better way to handle subordinate openstack upgrades: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1828422

[Corey Bryant (corey.bryant)]

Dmitrii said he'd be able to give this charm fix a test, so waiting to hear back. Thanks Dmitrii!

[Dmitrii Shcherbakov (dmitriis)]

Corey,

Ran into https://bugs.launchpad.net/charm-keystone/+bug/1827668 while re-deploying today.

I redeployed again and I am about to start upgrading the cloud.

I will post the results shortly.

[Dmitrii Shcherbakov (dmitriis)]

Tested with the patch from #19:

https://private-fileshare.canonical.com/~dima/charm-dumps/10-05-2019-var-log-nova-compute-kvm-1.tar.gz

dpkg -l | grep -P 'fwaas|neutron-l3-agent'
ii neutron-fwaas-common 1:13.0.1-0ubuntu1~cloud0 all Firewall-as-a-Service driver for OpenStack Neutron
ii neutron-l3-agent 2:13.0.2-0ubuntu3.1~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
ii python3-neutron-fwaas 1:13.0.1-0ubuntu1~cloud0 all Firewall-as-a-Service driver for OpenStack Neutron

Looks good to me.

[Dmitrii Shcherbakov (dmitriis)]

While #19 solves the problem for neutron-l3-agent we still need to poke neutron-openvswitch to run config-changed code because there may be version-specific code-paths in hooks/neutron_ovs_context.py which I am assuming will be tracked in https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1828422

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/658062
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=96b5e8a2493305ec7667c153d070dffda1bf547b
Submitter: Zuul
Branch: master

commit 96b5e8a2493305ec7667c153d070dffda1bf547b
Author: Corey Bryant <email address hidden>
Date: Thu May 9 08:13:51 2019 -0400

    py3: deal with more subordinate dependencies

    Ensure subordinate py3 packages are installed if their py2
    counter-parts are currently installed for neutron-openvswitch.
    In this case we had missed python(3)-neutron-fwaas.

    Change-Id: I9d5e7c2ad034cd7bdf9cbf292b9107577860dac1
    Closes-Bug: #1828259

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/19.04
Review: https://review.opendev.org/658404

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/658404
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=d33fa57b40994483cf88f338a4d000e06aaf6cc5
Submitter: Zuul
Branch: stable/19.04

commit d33fa57b40994483cf88f338a4d000e06aaf6cc5
Author: Corey Bryant <email address hidden>
Date: Thu May 9 08:13:51 2019 -0400

    py3: deal with more subordinate dependencies

    Ensure subordinate py3 packages are installed if their py2
    counter-parts are currently installed for neutron-openvswitch.
    In this case we had missed python(3)-neutron-fwaas.

    Change-Id: I9d5e7c2ad034cd7bdf9cbf292b9107577860dac1
    Closes-Bug: #1828259
    (cherry picked from commit 96b5e8a2493305ec7667c153d070dffda1bf547b)

[Edward Hope-Morley (hopem)]

I see to have just hit this for a fresh install of bionic+rocky:

root@crustle:~# dpkg -l| egrep "keepalived|neutron-l3"
ii keepalived 1:1.3.9-1ubuntu0.18.04.2 amd64 Failover and monitoring daemon for LVS clusters

root@crustle:~# tail -n 1 /var/log/neutron/neutron-l3-agent.log
2019-06-05 23:43:07.487 385426 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process.ProcessMonitor._check_child_processes" :: held 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:285

root@crustle:~# grep l3-agent /var/log/apt/history.log |grep -vi install
Commandline: apt-get --assume-yes purge keepalived neutron-l3-agent
Purge: keepalived:amd64 (1:1.3.9-1ubuntu0.18.04.2), neutron-l3-agent:amd64 (2:13.0.2-0ubuntu3.2~cloud0)

root@crustle:~# grep l3-agent /var/log/dpkg.log | grep -v status
2019-06-05 23:18:41 install neutron-l3-agent:all <none> 2:13.0.2-0ubuntu3.2~cloud0
2019-06-05 23:20:03 configure neutron-l3-agent:all 2:13.0.2-0ubuntu3.2~cloud0 <none>
2019-06-05 23:43:14 remove neutron-l3-agent:all 2:13.0.2-0ubuntu3.2~cloud0 <none>
2019-06-05 23:43:17 purge neutron-l3-agent:all 2:13.0.2-0ubuntu3.2~cloud0 <none>

[Edward Hope-Morley (hopem)]

using stable (19.04) charms ^^

[Edward Hope-Morley (hopem)]

Ok this was my mistake, i accidentally caused the neutron-openvswitch charm to be installed on the same host as the neutron-gateway (and n-ovs charm uninstalles l3-agent when dvr is not used). So not a bug after all.