Missing neutron-ovs-cleanup service

Bug #1249708 reported by Lorin Hochstein on 2013-11-10
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Medium
Unassigned
neutron (Ubuntu)
High
James Page

Bug Description

When the neutron network node is rebooted, the "neutron-ovs-cleanup" command must be run before the neutron services start or several services like L3 and DHCP won't work properly .

The RHEL OpenStack packages install a system service that automatically runs this command, but on Ubuntu the sysadmin is responsible for running this command manually or setting up a system service manually.

It would make sysadmin life easier if the Ubuntu packages also installed a service that invoked neutron-ovs-cleanup at the appropriate time.

References:

 * https://bugs.launchpad.net/openstack-manuals/+bug/1156861
* https://bugs.launchpad.net/neutron/+bug/1084355
 * https://ask.openstack.org/en/question/2605/

Related branches

James Page (james-page) on 2013-12-09
Changed in cloud-archive:
status: New → Triaged
importance: Undecided → Medium
Changed in neutron (Ubuntu):
status: New → Triaged
importance: Undecided → High
assignee: nobody → James Page (james-page)

related to bug 1091605

On Mon, Dec 9, 2013 at 5:28 PM, James Page <email address hidden> wrote:

> ** Changed in: cloud-archive
> Status: New => Triaged
>
> ** Changed in: cloud-archive
> Importance: Undecided => Medium
>
> ** Also affects: neutron (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: neutron (Ubuntu)
> Status: New => Triaged
>
> ** Changed in: neutron (Ubuntu)
> Importance: Undecided => High
>
> ** Changed in: neutron (Ubuntu)
> Assignee: (unassigned) => James Page (james-page)
>
> --
> You received this bug notification because you are subscribed to ubuntu-
> cloud-archive.
> Matching subscriptions: cloud archive
> https://bugs.launchpad.net/bugs/1249708
>
> Title:
> Missing neutron-ovs-cleanup service
>
> Status in Ubuntu Cloud Archive:
> Triaged
> Status in “neutron” package in Ubuntu:
> Triaged
>
> Bug description:
> When the neutron network node is rebooted, the "neutron-ovs-cleanup"
> command must be run before the neutron services start or several
> services like L3 and DHCP won't work properly .
>
> The RHEL OpenStack packages install a system service that
> automatically runs this command, but on Ubuntu the sysadmin is
> responsible for running this command manually or setting up a system
> service manually.
>
> It would make sysadmin life easier if the Ubuntu packages also
> installed a service that invoked neutron-ovs-cleanup at the
> appropriate time.
>
> References:
>
> * https://bugs.launchpad.net/openstack-manuals/+bug/1156861
> * https://bugs.launchpad.net/neutron/+bug/1084355
> * https://ask.openstack.org/en/question/2605/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-archive/+bug/1249708/+subscriptions
>

--
Cheers,
Jian

James Page (james-page) wrote :

I started working on this and I agree that this tool needs to be run; but for me its unclear as to when this should happen.

Does this need to happen on network forwarding nodes only (l3 and dhcp services) and for the openvswitch/ml2 plugins only? In which case it might make sense to run this in the pre-start block of the neutron-plugin-openvswitch-agent upstart configuration.

Lorin Hochstein (lorinh) wrote :

I believe it's an issue whenever you have a Neutron service that creates tap devices (openvswitch internal ports). I think only the l3 and dhcp services do this currently,

Jian Wen (wenjianhn) wrote :

I think this tool needs to be run on l3 nodes and dhcp nodes.
It should only be run when rebooting a node.
And it should finish the cleaning before l3 agent or dhcp agent starts.

In some environments, nova compute, l3 agent and dhcp agent are on the same node [1].
We need to make sure that the tool finishes the cleaning before nova compute starts.

It's kind of complicated. That's why I don't like the idea that using a upstart/init job to do the cleaning.
Maybe it's better that l3 agent and dhcp agent replug the ports themselves during starting.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1010941

James Page (james-page) on 2013-12-11
Changed in neutron (Ubuntu):
status: Triaged → In Progress
James Page (james-page) wrote :

I agree its complicated; I discussed with upstream PTL and its only needed for l3 and dhcp agents with the openvswitch plugin; the dhcp agent is also used by other plugins so the dependency on cleanup completing first needs to be soft (this can be done).

My current thing is to add this to the neutron-plugin-openvswitch-agent package, add conditional waits to the l3 and dhcp agent configurations and a hard wait for the neutron-plugin-openvswitch-agent.

We can also add the same soft wait to nova-compute if this is required - that creates tap devices as well so I think Jian is correct in #4.

George Shuklin (george-shuklin) wrote :

It should start on every compute and network node. It clear up all strange artefacts saved by ovs-vsctl.

How to get 'strange artifacts':
On any working compute/network node with active networking on instances perform any operations on ovs-vsctl, f.e.:
ovs-vsctl add-br just_empty_bridge
ovs-vsctl del-br just_empty_bridge

ovs will save all current tap/tun qr-* interfaces in br-int/br-ext bridges and restore them on next boot inside own config, even if they are not available at server start.

This will cause problems for existing instances during their startup.

neutron-ovs-cleanup should run during startup on all non-diskless nodes to ensure nothing strange is exists in bridges prior fists instance/agent start.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 1:2014.1~b2-0ubuntu2

---------------
neutron (1:2014.1~b2-0ubuntu2) trusty; urgency=medium

  * debian/patches/skip-lb-test.patch: Skipped lb configuration
    test.
 -- Chuck Short <email address hidden> Mon, 27 Jan 2014 12:01:50 -0500

Changed in neutron (Ubuntu):
status: In Progress → Fix Released
James Page (james-page) on 2015-03-24
Changed in cloud-archive:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.