neutron-ovs-cleanup runs on package upgrade

[James Page]

== Impact ==

Upgrades to OpenStack Ussuri will result in service disruption to the network datapath for instances and other virtual networking concepts.

== Test Case ==

Standalone
apt install neutron-openvswitch-agent
(add proposed pocket)
apt install neutron-openvswitch-agent
neutron-ovs-cleanup unit will be restarted

(UCA)
add-apt-repository cloud-archive:train
apt install neutron-openvswitch-agent
add-apt-repository cloud-archive:ussuri
apt dist-upgrade

(neutron-ovs-cleanup will be restarted)

Full cloud (UCA or distro)
Deploy OpenStack Train (using Juju charms)
Create networking, boot instances and confirm connectivity to instances
Upgrade to OpenStack Ussuri (using Juju charms)
Networking to instances will be lost

== Regression Potential ==

Medium - the fix is purely in package and ensures that the neutron-*-cleanup scripts are not started on install or restarted during an upgrade.

== Original Bug Report ==
Release: Focal/Bionic+Ussuri:

As part of the package upgrade for neutron-openvswitch-agent, the neutron-ovs-cleanup systemd unit is restarted.

This unit should only be run on reboot, and not during package upgrades.

It has the side effect of removing all ports from the local OVS bridge, impacting networking to services.

[Alex Kavanagh (ajkavanagh)]

Some additional information regarding how the bug was found:

During the upgrade the ovs runs, deleting the ports. They don't come back, even after restarting the services, which suggests an issue within the neutron openvswitch agent in terms of re-creating ports that have been deleted.

from the ovsdb-tool -mm show-log tool on the neutron-gateway AFTER the upgrade.

```
record 44: 2020-06-25 19:10:48.024
  table Interface row "tap03282320-fb" (74bf6e22):
    delete row
  table Interface row "tap7e15b3c6-e3" (d331a9b6):
    delete row
  table Interface row "tap8e76995d-a6" (f8ca1e54):
    delete row
  table Port row "tap03282320-fb" (d891c2a5):
    delete row
  table Port row "tap7e15b3c6-e3" (8f105882):
    delete row
  table Port row "tap8e76995d-a6" (b5f389e0):
    delete row
  table Bridge row "br-int" (56555638):
    ports=[704ecf79-ff42-44bd-a72c-c586f59f433b, dcfd1c64-2e07-4bf7-8815-a96950384705, f38673c2-87c9-4ce1-b160-236637caf8d2]
  table Open_vSwitch row 70200fb1 (70200fb1):
    next_cfg=111
```

i.e. it was asked to be deleted.

from syslog (during upgrade):

```
Jun 25 19:10:44 juju-ed722c-mojo-14 systemd[1]: Stopping Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:44 juju-ed722c-mojo-14 ovsdb-client: ovs|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Stopped Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Stopped OpenStack Neutron OVS cleanup.
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Stopping OpenStack Neutron OVS cleanup...
Jun 25 19:10:46 juju-ed722c-mojo-14 systemd[1]: Starting OpenStack Neutron OVS cleanup...
Jun 25 19:10:48 juju-ed722c-mojo-14 kernel: [ 6499.589055] device tap03282320-fb left promiscuous mode
Jun 25 19:10:48 juju-ed722c-mojo-14 kernel: [ 6499.611240] device tap7e15b3c6-e3 left promiscuous mode
Jun 25 19:10:48 juju-ed722c-mojo-14 kernel: [ 6499.612612] device tap8e76995d-a6 left promiscuous mode
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Started OpenStack Neutron OVS cleanup.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Starting Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Started Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Reloading.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopping Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopped Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopped OpenStack Neutron OVS cleanup.
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Stopping OpenStack Neutron OVS cleanup...
Jun 25 19:10:48 juju-ed722c-mojo-14 systemd[1]: Starting OpenStack Neutron OVS cleanup...
Jun 25 19:10:50 juju-ed722c-mojo-14 systemd[1]: Started OpenStack Neutron OVS cleanup.
Jun 25 19:10:50 juju-ed722c-mojo-14 systemd[1]: Starting Openstack Neutron Open vSwitch Plugin Agent...
Jun 25 19:10:50 juju-ed722c-mojo-14 systemd[1]: Started Openstack Neutron Open vSwitch Plugin Agent.
Jun 25 19:10:51 juju-ed722c-mojo-14 systemd[1]: Reloading.
Jun 25 19:10:51 juju-ed722c-mojo-...

[James Page (james-page)]

Updated packages upload for groovy and for focal (for SRU team review).

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:16.0.0-0ubuntu3

---------------
neutron (2:16.0.0-0ubuntu3) groovy; urgency=medium

  * Ensure OVS and linuxbridge cleanup systemd units are not restarted
    on upgrade or started on install (LP: #1885264):
    - d/rules: Pass --no-stop-on-upgrade to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Pass --no-start to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Also pass --no-restart-after-upgrade to deal with bug
      in older debhelper versions in Ubuntu Cloud Archive.
    - d/rules: Pass appropriate -X flags to each call to
      dh_install{systemd,init} to ensure that duplicate snippets are
      not generated in maintainer scripts.
  * d/p/bug1879717.patch: Cherry pick unit testing fix for updates into
    ovsdbapp.
  * d/rules,test-blacklist.txt: Blacklist misc failing tests on groovy
    until next set of snapshots.
  * d/p/misc-py38-p1.patch: Cherry pick fix for failing unit test.

 -- James Page <email address hidden> Thu, 02 Jul 2020 07:51:11 +0100

[Łukasz Zemczak (sil2100)]

Hello James, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.0.0-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

focal testing:

neutron-ovs-cleanup (on first install from update pocket)

     Active: active (exited) since Wed 2020-07-08 16:36:12 UTC; 32s ago

neutron-ovs-cleanup (after package upgrade to -proposed version)

     Active: active (exited) since Wed 2020-07-08 16:36:12 UTC; 1min 18s ago

task not restarted as part of the package upgrade.

[James Page (james-page)]

bionic-ussuri/proposed testing:

neutron-ovs-cleanup not restarted on package upgrade:

$ sudo apt install --reinstall neutron-openvswitch-agent
...

$ sudo systemctl status neutron-ovs-cleanup

● neutron-ovs-cleanup.service - OpenStack Neutron OVS cleanup
   Loaded: loaded (/lib/systemd/system/neutron-ovs-cleanup.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2020-07-10 12:56:07 UTC; 47min ago
 Main PID: 12976 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4703)
   CGroup: /system.slice/neutron-ovs-cleanup.service

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:17.0.0~git2020070709.792f4efce9-0ubuntu1~cloud0
---------------

 neutron (2:17.0.0~git2020070709.792f4efce9-0ubuntu1~cloud0) focal-victoria; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 neutron (2:17.0.0~git2020070709.792f4efce9-0ubuntu1) groovy; urgency=medium
 .
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/py3dist-overrides: Created with win32 dep wmi.
   * d/*.upstart.in: Dropped. No longer needed.
   * d/control: Update Standards-Version to 4.5.0.
   * d/p/fix-neutron-ipset-cleanup-tool.patch: Dropped. Fixed upstream.
   * d/p/bug1879717.patch: Dropped. Fixed upstream.
   * d/p/misc-py38-p1.patch: Dropped. Fixed upstream.
   * d/p/skip-iptest.patch: Rebased.
   * d/p/flake8-legacy.patch: Dropped. No longer needed.
   * d/neutron-common.install: Remove netns-cleanup.filters along with
     upstream removal as part of migration to oslo.privsep.
   * d/rules: Set OS_* env variables for unit test execution in line with
     upstream tox.ini.
 .
 neutron (2:16.0.0-0ubuntu3) groovy; urgency=medium
 .
   * Ensure OVS and linuxbridge cleanup systemd units are not restarted
     on upgrade or started on install (LP: #1885264):
     - d/rules: Pass --no-stop-on-upgrade to dh_install{systemd,init} for
       ovs and linuxbridge cleanup systemd units.
     - d/rules: Pass --no-start to dh_install{systemd,init} for
       ovs and linuxbridge cleanup systemd units.
     - d/rules: Also pass --no-restart-after-upgrade to deal with bug
       in older debhelper versions in Ubuntu Cloud Archive.
     - d/rules: Pass appropriate -X flags to each call to
       dh_install{systemd,init} to ensure that duplicate snippets are
       not generated in maintainer scripts.
   * d/p/bug1879717.patch: Cherry pick unit testing fix for updates into
     ovsdbapp.
   * d/rules,test-blacklist.txt: Blacklist misc failing tests on groovy
     until next set of snapshots.
   * d/p/misc-py38-p1.patch: Cherry pick fix for failing unit test.
 .
 neutron (2:16.0.0-0ubuntu2) groovy; urgency=medium
 .
   * d/control: Add net-tools dependency to python3-neutron for netstat
     (LP: #1881852).
   * d/neutron-common.install: Add rootwrap.d/netns-cleanup.filters
     (LP: #1881852).
   * d/p/fix-neutron-ipset-cleanup-tool.patch: Fix Traceback (LP: #1881771).
 .
 neutron (2:16.0.0-0ubuntu1) groovy; urgency=medium
 .
   * d/watch: Scope to 16.x series.
   * New upstream release for OpenStack Ussuri (LP: #1877642).
   * d/p/monkey-patch-original-current-thread.patch: Dropped. Fixed
     in upstream release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:16.0.0-0ubuntu0.20.04.2

---------------
neutron (2:16.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * Ensure OVS and linuxbridge cleanup systemd units are not restarted
    on upgrade or started on install (LP: #1885264):
    - d/rules: Pass --no-stop-on-upgrade to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Pass --no-start to dh_install{systemd,init} for
      ovs and linuxbridge cleanup systemd units.
    - d/rules: Also pass --no-restart-after-upgrade to deal with bug
      in older debhelper versions in Ubuntu Cloud Archive.
    - d/rules: Pass appropriate -X flags to each call to
      dh_install{systemd,init} to ensure that duplicate snippets are
      not generated in maintainer scripts.

 -- James Page <email address hidden> Thu, 02 Jul 2020 08:27:02 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.