neutron-ovs-cleanup failing to start with ovs bonding

Bug #1460494 reported by Ross Krumbeck
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
yalei wang

Bug Description

System is running Openstack Kilo:

[root@network01 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

[root@network01 ~]# uname -r
3.10.0-229.4.2.el7.x86_64

[root@network01 ~]# rpm -qa|grep neutron
openstack-neutron-ml2-2015.1.0-1.el7.noarch
openstack-neutron-common-2015.1.0-1.el7.noarch
openstack-neutron-openvswitch-2015.1.0-1.el7.noarch
python-neutronclient-2.3.11-1.el7.noarch
python-neutron-2015.1.0-1.el7.noarch
openstack-neutron-2015.1.0-1.el7.noarch

Issue is with python-neutron-2015.1.0-1.el7.noarch line 316 of:

/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py

get_vif_ports function is expecting that each ovs port will have an interface with a matching name, i.e. ovs port eth0 must have ovs Interface with name eth0. This does not work for bonded interfaces created within ovs as port bond2 may have Interface eth0 and Interface eth1 as an example.

2015-05-29 16:53:06.034 2833 INFO neutron.common.config [-] Logging enabled!
2015-05-29 16:53:06.039 2833 INFO neutron.common.config [-] /usr/bin/neutron-ovs-cleanup version 2015.1.0
2015-05-29 16:53:06.502 2833 ERROR neutron.agent.ovsdb.impl_vsctl [-] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'bond2'].
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl Traceback (most recent call last):
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl log_fail_as_error=False).rstrip()
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl raise RuntimeError(m)
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl RuntimeError:
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'bond2']
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl Exit code: 1
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl Stdin:
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl Stdout:
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl Stderr: ovs-vsctl: no row "bond2" in table Interface
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl
2015-05-29 16:53:06.502 2833 TRACE neutron.agent.ovsdb.impl_vsctl
2015-05-29 16:53:06.503 2833 CRITICAL neutron [-] RuntimeError:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'bond2']
Exit code: 1
Stdin:
Stdout:
Stderr: ovs-vsctl: no row "bond2" in table Interface

2015-05-29 16:53:06.503 2833 TRACE neutron Traceback (most recent call last):
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/bin/neutron-ovs-cleanup", line 10, in <module>
2015-05-29 16:53:06.503 2833 TRACE neutron sys.exit(main())
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/ovs_cleanup.py", line 100, in main
2015-05-29 16:53:06.503 2833 TRACE neutron ports = collect_neutron_ports(available_configuration_bridges)
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/ovs_cleanup.py", line 60, in collect_neutron_ports
2015-05-29 16:53:06.503 2833 TRACE neutron ports += [port.port_name for port in ovs.get_vif_ports()]
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 316, in get_vif_ports
2015-05-29 16:53:06.503 2833 TRACE neutron check_error=True)
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 139, in db_get_val
2015-05-29 16:53:06.503 2833 TRACE neutron check_error=check_error)
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 83, in execute
2015-05-29 16:53:06.503 2833 TRACE neutron txn.add(self)
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 68, in __exit__
2015-05-29 16:53:06.503 2833 TRACE neutron self.result = self.commit()
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 50, in commit
2015-05-29 16:53:06.503 2833 TRACE neutron res = self.run_vsctl(args)
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 70, in run_vsctl
2015-05-29 16:53:06.503 2833 TRACE neutron ctxt.reraise = False
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-29 16:53:06.503 2833 TRACE neutron six.reraise(self.type_, self.value, self.tb)
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl
2015-05-29 16:53:06.503 2833 TRACE neutron log_fail_as_error=False).rstrip()
2015-05-29 16:53:06.503 2833 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
2015-05-29 16:53:06.503 2833 TRACE neutron raise RuntimeError(m)
2015-05-29 16:53:06.503 2833 TRACE neutron RuntimeError:
2015-05-29 16:53:06.503 2833 TRACE neutron Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'bond2']
2015-05-29 16:53:06.503 2833 TRACE neutron Exit code: 1
2015-05-29 16:53:06.503 2833 TRACE neutron Stdin:
2015-05-29 16:53:06.503 2833 TRACE neutron Stdout:
2015-05-29 16:53:06.503 2833 TRACE neutron Stderr: ovs-vsctl: no row "bond2" in table Interface
2015-05-29 16:53:06.503 2833 TRACE neutron
2015-05-29 16:53:06.503 2833 TRACE neutron

ovs_lib.py should be updated so that for an ovs bond port all interfaces belonging to the bond are looked up without relying on the name of the Interface to be matching.

Ross Krumbeck (rkrum)
description: updated
Revision history for this message
Ross Krumbeck (rkrum) wrote :

A work around is to configure standard linux bonding, then add linux bond interface as an ovs port (rather than creating bond within openvswitch).

description: updated
yalei wang (yalei-wang)
Changed in neutron:
assignee: nobody → yalei wang (yalei-wang)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/189568

Changed in neutron:
status: New → In Progress
Kyle Mestery (mestery)
Changed in neutron:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/203992

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/189568
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b4e42a341894177a1c870a48b2641bae6c36696e
Submitter: Jenkins
Branch: master

commit b4e42a341894177a1c870a48b2641bae6c36696e
Author: Yalei Wang <email address hidden>
Date: Tue Jun 9 13:46:57 2015 +0800

    Add new ovs DB API to inquire interfaces name list in a bridge

    In OVS, ports don't equal to interfaces when a bond port created. This patch
    add the new API get_iface_name_list to get the interfaces' name, and it's
    supplementary to the current get_port_name_list API.

    Change-Id: I29c220e099b8dcf78248e2d660c435578bb2932d
    Partial-Bug: #1460494

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/207903

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (70.7 KiB)

Reviewed: https://review.openstack.org/207903
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9badcd249dab2d3330f0cd608496b59c9f44499a
Submitter: Jenkins
Branch: feature/pecan

commit 991ea00e6c115343eabecc62e86072175823f81f
Author: Moshe Levi <email address hidden>
Date: Thu Jul 30 12:04:15 2015 +0300

    SR-IOV: Fix SR-IOV agent to run ip link commands as root

    Commit https://review.openstack.org/#/c/155523/
    remove the remaining root_helper args, but didn't
    update the SR-IOV agent to execute them as root.

    This patch updates the agent to execute ip link commands
    as root and pass options argument as a list in the
    self._as_root method.

    Closes-Bug: #1479694
    Change-Id: I53cafd61845a69fae3a759fb7526950d655ffa20

commit 5b3bacedf6c014815bef03c2e821b5eb8ef92dcb
Author: Saksham Varma <email address hidden>
Date: Thu Jul 23 22:46:23 2015 -0700

    Moved out cisco n1kv mech driver and db models

    Moving out Cisco N1Kv database models and the mech driver
    to the openstack/networking-cisco repo as part of the second
    phase vendor-core decomposition

    Partial-Bug: #1479123
    Partial-implements: blueprint core-vendor-decomposition

    Change-Id: I65a704b238d8cbe2951a6912fa4f1e8518c6f412

commit 65ac8cd0a80548e115b8ec1d4cfd47a94422fddf
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Jul 29 20:44:01 2015 +0000

    Updated from global requirements

    Change-Id: I3a884a73d51df0f93b5cc88b3e3250d81bb1f455

commit f0e8356d04d04600c072a1c0f0bdf274ff19ab8c
Author: sridhargaddam <email address hidden>
Date: Mon Jul 27 03:46:48 2015 +0000

    Update dhcp agent cache for network:dhcp ports

    When a network with a dhcp_enabled subnet is scheduled on a dhcp
    agent, dhcp driver creates the network:dhcp port for the subnet.
    However, the port info is not updated in dhcp agents internal cache.
    Subsequently if the user deletes the network:dhcp port, the port is
    properly deleted on the server side (i.e., in the database) and when
    the port_delete_end notification is sent to the dhcp agent, it simply
    ignores it as the port entry would be missing in the cache. This patch
    fixes this issue by updating the dhcp agents cache when dhcp driver
    creates the network:dhcp port for the subnets.

    Closes-Bug: #1478426
    Change-Id: I69f5834dd964a4320c606c4e0aa2cdba70416943

commit cb60d0bb4e0cc0cba68f59fdf5f4e89d6ec52950
Author: changzhi <email address hidden>
Date: Thu Jul 16 10:14:16 2015 +0800

    Keep dns nameserver order consistency

    Currently, there is no dns servers prioritization for subnets
    for Neutron.

    Generally speaking, it is useful to keep the order of dns
    nameservers consistent. Add a new column named 'order' in table
    'dnsnameservers' and add nameserver into DB one by one.

    Closes-Bug: #1218629
    Implements: blueprint keep-dns-nameserver-orderconsistency
    Change-Id: Id937aea411397d39370368a4eb45be26c4eefa9e

commit b39e1469e824bc8bc79e1ecafa98825a94811c0b
Author: Salvatore Orlando <email address hidden>
Date: Tue Jun 23 04:54:2...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by yalei wang (<email address hidden>) on branch: master
Review: https://review.openstack.org/203992
Reason: https://review.openstack.org/#/c/200279/

fixed it.

Assaf Muller (amuller)
Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.