[SRU] neutron-ovs-cleanup runs earlier than ovsdb-server when there are many port

[Seyeong Kim]

[Impact]

In code, Basically all ubuntu version could be affected.
I'm trying to do test them

neutron-ovs-cleanup service runs earlier than ovsdb-server sometimes.

then, neutron-ovs-cleanup service's status remains failed
with below error msg

"ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)"

that is fine for booting.

but.

If someday upgrade pkgs related to neutron, it runs neutron-ovs-cleanup again because it is not in active status.

eventually, neutron-ovs-cleanup deletes all ports on running compute nodes.

related logs here https://paste.ubuntu.com/p/QkkwP2vPDv/

[Test Case]

1. deploy openstack env
2. run below scripts to add many ports
#!/bin/bash

juju ssh nova-compute/0 '
sudo modprobe dummy
for i in {1..1000}
do
  echo $i
  sudo ip link add dummy$i type dummy;sudo ip link set name t-test$i dev dummy$i;sudo ovs-vsctl add-port br-int t-test$i
done
'
3. systemctl list-units --failed
- check if there is failed neutron-ovs-cleanup service

[Regression]
Changes dependency on debian/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in
need to reboot if this patch is affected. If this symptom is not affected your env. no need to reboot.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1752838_xenial.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Hi Mathew Hodson, In the future can you please explain your reasoning when you set bug importance? Thanks.

[Corey Bryant (corey.bryant)]

Hi Seyeong, I think your patch makes sense. I wonder if we also need to add an After statement to ensure openvswitch-switch starts before neutron-ovs-cleanup:

+Requires=openvswitch-switch.service
+After=openvswitch-switch.service

Note that the LSB init info in the beginning of /etc/init.d/neutron-ovs-cleanup does have a "Required-Start: ... openvswitch-switch", but that is apparently not enough, or perhaps needs to be replaced by your patch.

I chatted with dosaboy a bit about this and it seems like we should also review nova-compute, neutron-l3-agent, and neutron-dhcp-agent as part of this fix. They need to start after neutron-ovs-cleanup is finished and they use "Should-Start: ... neutron-ovs-cleanup" in their LSB init info.

[Corey Bryant (corey.bryant)]

Current behavior: starting neutron-ovs-cleanup when openvswitch-switch is stopped: https://paste.ubuntu.com/p/MxHT5rXJJ2/

New behavior [1]: starting neutron-ovs-cleanup when openvswitch-switch is stopped: https://paste.ubuntu.com/p/wzJ3kh6CPp/

[1] After updating /lib/systemd/system/neutron-ovs-cleanup.service with:
Description=OpenStack Neutron OVS cleanup
+Requires=openvswitch-switch.service
+After=openvswitch-switch.service

[Corey Bryant (corey.bryant)]

Note: this affects back to kilo. It doesn't affect icehouse as we were fully using upstart at that point.

[Corey Bryant (corey.bryant)]

Correction. kilo is trusty only at this point so kilo is not affected.

[Corey Bryant (corey.bryant)]

This affects xenial-mitaka and above. My test results above are on xenial-queens.

[Corey Bryant (corey.bryant)]

Triaged high as this neutron-ovs-cleanup init scripts are not behaving as expected, which may cause a number of issues.

[Corey Bryant (corey.bryant)]

I reviewed the systemd unit files that have a dependency on neutron-ovs-cleanup and I think we could use some updates there as well to tighten up the dependency on neutron-ovs-cleanup.

We currently have the following and I think the latter 3 should be updated with a Requires like neutron-openvswitch-agent.service has. The After= only handles order when unit files are started together, whereas Requires= will start the required service if it is not already active and will fail this serivce if the required service fails to start.

/lib/systemd/system/neutron-openvswitch-agent.service
-----------------------------------------------------
[Unit]
Description=Openstack Neutron Open vSwitch Plugin Agent
After=mysql.service postgresql.service rabbitmq-server.service keystone.service
Requires=neutron-ovs-cleanup.service

/lib/systemd/system/neutron-l3-agent.service
--------------------------------------------
[Unit]
Description=OpenStack Neutron L3 agent
After=mysql.service postgresql.service rabbitmq-server.service keystone.service openvswitch-switch.service neutron-ovs-cleanup.service

/lib/systemd/system/neutron-dhcp-agent.service
----------------------------------------------
[Unit]
Description=OpenStack Neutron DHCP agent
After=mysql.service postgresql.service rabbitmq-server.service keystone.service neutron-ovs-cleanup.service

/lib/systemd/system/nova-compute.service
----------------------------------------
[Unit]
Description=OpenStack Compute
After=libvirtd.service postgresql.service mysql.service keystone.service rabbitmq-server.service ntp.service neutron-ovs-cleanup.service

Another point of concern was that neutron-ovs-cleanup may not finish prior to dependent units starting. That won't happen with the After= option specified. From the systemd.service man page:
"Note that systemd will consider the unit to be in the state "starting" until the program has terminated, so ordered dependencies will wait for the program to finish before starting themselves."

neutron-ovs-cleanup is a Type=oneshot unit with RemainAfterExit=yes, which means that it will run

[Corey Bryant (corey.bryant)]

... it will run it's action without keeping an active process. RemainAfterExit=yes causes the unit to remain active after the start action exited successfully.

[Seyeong Kim (seyeongkim)]

Hello

I've been testing xenial-newton env if there is also same issue.
but final symptom was not same.

it keep restarting on boot and eventually status is settled on active.
(there was error about connecting to ovsdb-server several times but.. )

I was stuck on that to find why it is different. not found yet.

Basically, yes. i agree with you. they need to be changed.

I'm going to check all of them. and will upload it today later.

Thanks.

[Seyeong Kim (seyeongkim)]

@corey,

I inserted service file in debian folder.
Tested with manipulated init.in file but no luck.

is it proper to upload nova debdiff here?

[Corey Bryant (corey.bryant)]

Adding Requires=neutron-ovs-cleanup.service to neutron-l3-agent.service, neutron-dhcp-agent.service, and nova-compute.service is turning out to be perhaps too tightly coupled.

Doing this would also require the binary packages for each of the above named services to depend on the neutron-openvswitch-agent binary package. This is because neutron-ovs-cleanup is installed by the neutron-openvswitch-agent binary package. If we add Requires=neutron-ovs-cleanup without adding the binary package dependency, then at install time the above named services would fail with a dependency error if neutron-ovs-cleanup is not already installed.

I just did some reboot testing and I think we are ok with just the After=neutron-ovs-cleanup for these services. I rebooted 10 times and each time neutron-l3-agent and neutron-dhcp-agent started after neutron-ovs-cleanup. So I believe we are already covered for reboots.

What this doesn't cover is service start order at package install time. neutron-l3-agent and neutron-dhcp-agent can be installed and started before neutron-ovs-cleanup is installed. In this case neutron-ovs-cleanup would start and run after neutron-l3-agent and neutron-dhcp-agent are started. I don't think this is a big deal at install time.

As for upgrades, neutron-ovs-cleanup doesn't restart, I believe because it is Type=oneshot. So that is very good news. You can 'apt dist-upgrade' to all new neutron packages and neutron-ovs-cleanup won't get restarted.

I think we should just move forward with the original issue reported by this bug and add AFter= and Requires= to neutron-ovs-cleanup.

[Corey Bryant (corey.bryant)]

Hi Seyeong, I've uploaded patched versions of neutron to xenial, xenial-newton, xenial-ocata, artful, and bionic. xenial, artful, and bionic are in the unapproved queue awaiting sru review.

Thanks again for the bug/patch.

Corey

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:12.0.0-0ubuntu1.1

---------------
neutron (2:12.0.0-0ubuntu1.1) bionic; urgency=medium

  * d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
    Adding dependency on openvswitch-switch.service because
    neutron-ovs-cleanup will fail if it is run earlier than it
    (LP: #1752838).

 -- Seyeong Kim <email address hidden> Thu, 01 Mar 2018 17:19:08 -0800

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

for xenial-ocata,

I ran script as above (changed after some more tests)

before installing patch, neutron openvswitch agent ran earlier than ovsdb-server(openvswitch-switch service)

after installing patch, openvswitch-switch runs earlier.

the symptom as described above is sometimes happening not all the times
but this patch is for possible issue, so I make this verification done

dpkg info

ii neutron-common 2:10.0.4-0ubuntu2~cloud1 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:10.0.4-0ubuntu2~cloud1 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:10.0.4-0ubuntu2~cloud1 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:10.0.4-0ubuntu2~cloud1 all Neutron is a virtual network service for Openstack - Python library

[Seyeong Kim (seyeongkim)]

for xenial-newton

I could see above error without patch, after running script and rebooting

after patch, order is correct and error is gone.

Thanks.

dpkg info

ii neutron-common 2:9.4.1-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:9.4.1-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:9.4.1-0ubuntu1~cloud2 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:9.4.1-0ubuntu1~cloud2 all Neutron is a virtual network service for Openstack - Python library

[Seyeong Kim (seyeongkim)]

for xenial-queens

After patching neutron-ovs-cleanup is following after ovsdb-server properly.

while testing xenial-queens

I noticed that neutron-openvswitch-agent has Requires, but not After for neutron-ovs-cleanup

and in syslog

neutron-openvswitch-agent started earlier than above two(ovsdb-server and neutron-ovs-cleanup) got error and had restarted several time(eventually it is fine)

We need to add this for neutron-openvswitch-agent for all releases?

or It is ok for now?

I'll change verification tag after discussion.

Thanks.

[Corey Bryant (corey.bryant)]

Based on Seyeong's testing input, I've uploaded new versions of neutron to all affected releases. These versions add After=neutron-ovs-cleanup to neutron-openvswitch-agent. Thanks Seyeong!

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

for xenial queens

updated pkg after last comment above,

now services run as below order

1. openvswitch-switch
2. neutron-ovs-cleanup
3. neutron-openvswitch-agent

this is expected

Thanks.

dpkg -l | grep neutron

ii neutron-common 2:12.0.0-0ubuntu1.2~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:12.0.0-0ubuntu1.2~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:12.0.0-0ubuntu1.2~cloud0 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:12.0.0-0ubuntu1.2~cloud0 all Neutron is a virtual network service for Openstack - Python library

[Seyeong Kim (seyeongkim)]

for xenial ocata

ok, service starting order is same as above(queens)

there is no error in neutron or openvswitch services with 1000ports

Thanks

dpkg -l | grep neutron

ii neutron-common 2:10.0.4-0ubuntu2~cloud1.1 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:10.0.4-0ubuntu2~cloud1.1 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:10.0.4-0ubuntu2~cloud1.1 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:10.0.4-0ubuntu2~cloud1.1 all Neutron is a virtual network service for Openstack - Python library

[Seyeong Kim (seyeongkim)]

for xenial newton

this is the same as above result,

in this case, I saw same issue as description(several connection error to ovsdb-server)

after patching, it has gone.

order is as expected.

Thanks

dpkg -l | grep neutron

ii neutron-common 2:9.4.1-0ubuntu1~cloud2.1 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:9.4.1-0ubuntu1~cloud2.1 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:9.4.1-0ubuntu1~cloud2.1 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:9.4.1-0ubuntu1~cloud2.1 all Neutron is a virtual network service for Openstack - Python library

[Łukasz Zemczak (sil2100)]

Hello Seyeong, or anyone else affected,

Accepted neutron into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:11.0.2-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Seyeong, or anyone else affected,

Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

for xenial

I checked that this patch is working.

1. deploy xenial
2. ran script on test case
3. reboot, and check if there is issue on order or failed service
4. upgrade to -proposed pkg
5. reboot, and check if there is issue(it is gone)

Thanks a lot

dpkg -l | grep neutron

ii neutron-common 2:8.4.0-0ubuntu7.1 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:8.4.0-0ubuntu7.1 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
rc neutron-plugin-openvswitch-agent 2:8.4.0-0ubuntu6 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:8.4.0-0ubuntu7.1 all Neutron is a virtual network service for Openstack - Python library

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

for xenial pike

I did same test on pike deployment

1. deploy pike env
2. run script
3. reboot nova-compute node and check if there is order weirdness
4. upgrading neutron pkgs from -proposed
5. reboot nova-compute node and check if it is correct order ( it is fine )

it is working fine.

Thanks.

dpkg -l | grep neutron
root@maas-node-05:~# dpkg -l | grep neutron
ii neutron-common 2:11.0.2-0ubuntu1.3~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:11.0.2-0ubuntu1.3~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:11.0.2-0ubuntu1.3~cloud0 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:11.0.2-0ubuntu1.3~cloud0 all Neutron is a virtual network service for Openstack - Python library

[Łukasz Zemczak (sil2100)]

Could this also be verified for artful? We can't release the xenial version if the artful counterpart isn't verified as well.

[Seyeong Kim (seyeongkim)]

hey sil2100,

sure, I'm trying to verify this on artful as well

with juju openstack deployment with artful release, there is an issue on deployment

and now im in sprint, so it is delaying..

I'm going to fix it or find new reproducer for this and verify this ASAP

Thanks.

[Seyeong Kim (seyeongkim)]

Hello

I verified this.

There was an issue about deploying lxd on artful machine.
I just remove them from lxd and depoyed with virtual machines each.

and done steps below

1. deploy artful openstack env ( without any lxd instance )
2. run scripts on test case
3. reboot compute node and check if there is abnormal order about openvswitch services and neutron services. ( wrong order there )
4. upgrade neutron pkgs
5. reboot compute node and check if there is mixed order, ( good. ovsdb-server -> cleanup -> agent )

Thanks a lot

dpkg -l | grep neutron

ii neutron-common 2:11.0.2-0ubuntu1.3 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:11.0.2-0ubuntu1.3 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
rc neutron-plugin-openvswitch-agent 2:11.0.2-0ubuntu1.1 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:11.0.2-0ubuntu1.3 all Neutron is a virtual network service for Openstack - Python library

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:11.0.2-0ubuntu1.3

---------------
neutron (2:11.0.2-0ubuntu1.3) artful; urgency=medium

  * d/neutron-openvswitch-agent.service.in: Ensure
    neutron-openvswitch-agent starts after neutron-ovs-cleanup.
  * d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
    Adding dependency on openvswitch-switch.service because
    neutron-ovs-cleanup will fail if it is run earlier than it
    (LP: #1752838).

 -- Seyeong Kim <email address hidden> Thu, 01 Mar 2018 17:19:08 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu7.1

---------------
neutron (2:8.4.0-0ubuntu7.1) xenial; urgency=medium

  * d/neutron-openvswitch-agent.service.in: Ensure
    neutron-openvswitch-agent starts after neutron-ovs-cleanup.
  * d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
    Adding dependency on openvswitch-switch.service because
    neutron-ovs-cleanup will fail if it is run earlier than it
    (LP: #1752838).

 -- Seyeong Kim <email address hidden> Thu, 01 Mar 2018 17:19:08 -0800

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:11.0.2-0ubuntu1.3~cloud0
---------------

 neutron (2:11.0.2-0ubuntu1.3~cloud0) xenial-pike; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:11.0.2-0ubuntu1.3) artful; urgency=medium
 .
   * d/neutron-openvswitch-agent.service.in: Ensure
     neutron-openvswitch-agent starts after neutron-ovs-cleanup.
   * d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
     Adding dependency on openvswitch-switch.service because
     neutron-ovs-cleanup will fail if it is run earlier than it
     (LP: #1752838).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:10.0.4-0ubuntu2~cloud1.1
---------------

 neutron (2:10.0.4-0ubuntu2~cloud1.1) xenial-ocata; urgency=medium
 .
   * d/neutron-openvswitch-agent.service.in: Ensure
     neutron-openvswitch-agent starts after neutron-ovs-cleanup.
   * d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
     Adding dependency on openvswitch-switch.service because
     neutron-ovs-cleanup will fail if it is run earlier than it
     (LP: #1752838).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:9.4.1-0ubuntu1~cloud2.1
---------------

 neutron (2:9.4.1-0ubuntu1~cloud2.1) xenial-newton; urgency=medium
 .
   * d/neutron-openvswitch-agent.service.in: Ensure
     neutron-openvswitch-agent starts after neutron-ovs-cleanup.
   * d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
     Adding dependency on openvswitch-switch.service because
     neutron-ovs-cleanup will fail if it is run earlier than it
     (LP: #1752838).

[Edward Hope-Morley (hopem)]

WARNING: this update has introduced a race between ovs-cleanup and neutron-openvswitch-agent such that when ovs is restarted, neutron-openvswitch-agent AND neutron-ovs-cleanup are also restarted resulting in the guest flows never getting re-added to br-tun. Recommend we either fix or pull this from the archive asap.

[Edward Hope-Morley (hopem)]

reverting in https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1758411

[Corey Bryant (corey.bryant)]

This change has been reverted due to regression. Please see https://bugs.launchpad.net/cloud-archive/mitaka/+bug/1758411 for more details.

[Seyeong Kim (seyeongkim)]

Require=openvswitch-switch.service in neutron-ovs-cleanup.service file

trigger calling neutron-ovs-cleanup when restarting openvswitch-switch

as earlier analysis, neutron-ovs-cleanup shouldn't have called while running machine.

it is only for booting sequence.

I'm testing it without require option.
There is issue error 14 alarm clock when calling neutron-ovs-cleanup on boot.

Keep analyzing

[Seyeong Kim (seyeongkim)]

Confirmed that removing Require-openvswitch-switch.server line in neutron-ovs-cleanup.service file solve regression and this order issue.

but as I mentioned just before, if there are a lot of (1000) ports there, 14 alarm clock error when running neutron-ovs-cleanup

and this[1] is pop up (this one I could see in the beginning sometimes)
also adding ExecPre= line solve it as well. ( in xenial )

[1] https://bugs.launchpad.net/cloud-archive/+bug/1630968

[Corey Bryant (corey.bryant)]

It seems that adding:

After=openvswitch-switch.service

to neutron-ovs-cleanup.service will be enough to fix this bug.

As long as openvswitch-switch is not already installed at the time of neutron-openvswitch-agent being installed, then neutron-openvswitch-agent will cause openvswitch-switch to be installed and neutron-ovs-cleanup should be forced to start after openvswitch-switch.

I'm not following your comment on 1630968. Is that issue related to this change?

[Seyeong Kim (seyeongkim)]

Hey Corey

yes I meant that yours. ( removing Requires line from regression patch )

1630968 is the other issue I faced.

not related to this issue. ( just mentioned it for log )

[Corey Bryant (corey.bryant)]

Thanks Seyeong. I've uploaded new versions of neutron with your patch (updated changelog as we discussed) for all affected releases. Artful and Xenial are awaiting review by the SRU team.

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:12.0.0-0ubuntu2

---------------
neutron (2:12.0.0-0ubuntu2) bionic; urgency=medium

  * d/neutron-openvswitch-agent.service.in,
    d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
    Ensure neutron-ovs-cleanup runs after openvswitch-switch and
    neutron-openvswitch-agent runs after neutron-ovs-cleanup (LP: #1752838).

 -- Seyeong Kim <email address hidden> Wed, 28 Mar 2018 04:44:38 -0700

[Seyeong Kim (seyeongkim)]

Hello,

for xenial-ocata

I followed below step to verify this.

1. deploy openstack xenial-ocata
2. source novarc
3. create instance ( https://pastebin.ubuntu.com/p/gZ3XXy9j4F/ )
- check this instance's host
4. login to host
5. adding dummy ports via ( https://paste.ubuntu.com/p/X35H85XWPh/ )
6. reboot and check if order is ok ( not ok here )
7. upgrade pkg
8. reboot and check if order is ok ( ok here )
9. regression check ( checking ovs-vsctl output if the port for #3 instance is exist ) on host
- service openvswitch-switch restart
- service neutron-openvswitch-agent restart
- check if port are still there

Thanks

dpkg -l | grep neutron
ii neutron-common 2:10.0.4-0ubuntu2~cloud2 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:10.0.4-0ubuntu2~cloud2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:10.0.4-0ubuntu2~cloud2 all Transitional package for neutron-openvswitch-agent

[Seyeong Kim (seyeongkim)]

Hello,

for xenial-newton

I followed below step to verify this. ( same as above )

1. deploy openstack xenial-ocata
2. source novarc
3. create instance ( https://pastebin.ubuntu.com/p/gZ3XXy9j4F/ )
- check this instance's host
4. login to host
5. adding dummy ports via ( https://paste.ubuntu.com/p/X35H85XWPh/ )
6. reboot and check if order is ok ( not ok here )
7. upgrade pkg
8. reboot and check if order is ok ( ok here )
9. regression check ( checking ovs-vsctl output if the port for #3 instance is exist ) on host
- service openvswitch-switch restart
- service neutron-openvswitch-agent restart
- check if port are still there

Thanks

dpkg -l | grep neutron
ii neutron-common 2:9.4.1-0ubuntu1~cloud3 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:9.4.1-0ubuntu1~cloud3 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:9.4.1-0ubuntu1~cloud3 all Transitional package for neutron-openvswitch-agent

[James Page (james-page)]

Hello Seyeong, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

Hello,

for xenial-queens

I followed below step to verify this. ( slightly different )

1. deploy openstack xenial-queens
2. create networks routers etc on horizon
3. create instance on horizon
4. login to host
5. adding dummy ports via ( https://paste.ubuntu.com/p/X35H85XWPh/ )
6. reboot and check if order is ok ( not ok here, at this time, systemd status failed )
7. upgrade pkg
8. reboot and check if order is ok ( ok here, clean )
9. regression check ( checking ovs-vsctl output if the port for #3 instance is exist ) on host
- service openvswitch-switch restart
- service neutron-openvswitch-agent restart
- check if port are still there

Thanks

dpkg -l | grep neutron
ii neutron-common 2:12.0.0-0ubuntu2~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:12.0.0-0ubuntu2~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:12.0.0-0ubuntu2~cloud0 all Transitional package for neutron-openvswitch-agent

[Łukasz Zemczak (sil2100)]

Hello Seyeong, or anyone else affected,

Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu7.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

Hello,

for xenial

I followed below step to verify this.

1. deploy openstack xenial
2. source novarc
3. create instance ( https://pastebin.ubuntu.com/p/gZ3XXy9j4F/ )
- check this instance's host
4. login to host
5. adding dummy ports via ( https://paste.ubuntu.com/p/X35H85XWPh/ )
6. reboot and check if order is ok ( not ok here )
7. upgrade pkg
8. reboot and check if order is ok ( ok here )
9. regression check ( checking ovs-vsctl output if the port for #3 instance is exist ) on host
- service openvswitch-switch restart
- service neutron-openvswitch-agent restart
- check if port are still there

Thanks

dpkg -l | grep neutron
ii neutron-common 2:8.4.0-0ubuntu7.3 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:8.4.0-0ubuntu7.3 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:8.4.0-0ubuntu7.3 all Neutron is a virtual network service for Openstack - Python library

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu7.3

---------------
neutron (2:8.4.0-0ubuntu7.3) xenial; urgency=medium

  [ Edward Hope-Morley ]
  * d/p/Wait-for-ovsdb_monitor-to-be-active-before-use-it.patch: backport
    fix to ensure ovsdb mon blocks on start until it is ready (LP: #1584647)

  [ Seyeong Kim ]
  * d/neutron-openvswitch-agent.service.in,
    d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
    Ensure neutron-ovs-cleanup runs after openvswitch-switch and
    neutron-openvswitch-agent runs after neutron-ovs-cleanup (LP: #1752838).

 -- Corey Bryant <email address hidden> Wed, 28 Mar 2018 11:42:05 -0400

[Łukasz Zemczak (sil2100)]

Hello Seyeong, or anyone else affected,

Accepted neutron into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:11.0.3-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

Hello,

for artful, ( without lxd )

I followed below step to verify this.

1. deploy openstack based on artful
2. source novarcv3_project
3. create instance via horizon
- check this instance's host
4. login to host
5. adding dummy ports via ( https://paste.ubuntu.com/p/X35H85XWPh/ )
6. reboot and check if order is ok ( not ok here, order is mixed )
7. upgrade pkg
8. reboot and check if order is ok ( ok here )
9. regression check ( checking ovs-vsctl output if the port for #3 instance is exist ) on host
- service openvswitch-switch restart
- service neutron-openvswitch-agent restart
- check if port are still there ( with ping )

Thanks

dpkg -l | grep neutron
ii neutron-common 2:11.0.3-0ubuntu2 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:11.0.3-0ubuntu2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:11.0.3-0ubuntu2 all Neutron is a virtual network service for Openstack - Python library

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:11.0.3-0ubuntu2

---------------
neutron (2:11.0.3-0ubuntu2) artful; urgency=medium

  * d/neutron-openvswitch-agent.service.in,
    d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
    Ensure neutron-ovs-cleanup runs after openvswitch-switch and
    neutron-openvswitch-agent runs after neutron-ovs-cleanup (LP: #1752838).

 -- Seyeong Kim <email address hidden> Wed, 28 Mar 2018 04:44:38 -0700

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted neutron into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Seyeong, Can you test pike-proposed again please? The updated package wasn't in proposed.

[Seyeong Kim (seyeongkim)]

Hello,

for pike

I followed below step to verify this.

1. deploy openstack xenial
2. source novarc
3. create instance ( https://pastebin.ubuntu.com/p/gZ3XXy9j4F/ )
- check this instance's host
4. login to host
5. adding dummy ports via ( https://paste.ubuntu.com/p/X35H85XWPh/ )
6. reboot and check if order is ok ( not ok here, order is mixed )
7. upgrade pkg
8. reboot and check if order is ok ( ok here )
9. regression check ( checking ovs-vsctl output if the port for #3 instance is exist ) on host
- service openvswitch-switch restart
- service neutron-openvswitch-agent restart
- check if port are still there

Thanks

ii neutron-common 2:11.0.3-0ubuntu2~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-openvswitch-agent 2:11.0.3-0ubuntu2~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-plugin-openvswitch-agent 2:11.0.3-0ubuntu2~cloud0 all Transitional package for neutron-openvswitch-agent
ii python-neutron 2:11.0.3-0ubuntu2~cloud0 all Neutron is a virtual network service for Openstack - Python library

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:11.0.3-0ubuntu2~cloud0
---------------

 neutron (2:11.0.3-0ubuntu2~cloud0) xenial-pike; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:11.0.3-0ubuntu2) artful; urgency=medium
 .
   * d/neutron-openvswitch-agent.service.in,
     d/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in:
     Ensure neutron-ovs-cleanup runs after openvswitch-switch and
     neutron-openvswitch-agent runs after neutron-ovs-cleanup (LP: #1752838).

[Corey Bryant (corey.bryant)]

This was already fix released for ocata but was buried under another version in the changelog. Manually marking fix released.

[Corey Bryant (corey.bryant)]

This was also already fix released for newton but was buried under another version in the changelog. Manually marking fix released.