"Executable not found: brctl" when using linuxbridge

[Ahmad Al-Shishtawy]

After failing to get openvswitch to connect to the external network (Internet) I decided to give linuxbridge a try. But it only got worse! Networking was not working at all.

After checking log files I found the following error in /var/log/kolla/neutron/neutron-linuxbridge-agent.log in the heka container.
I'm deploying on 10 servers 3 of them are control and network using (centos,source) to build images.

...
...
...
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent bridge_device = bridge_lib.BridgeDevice.addbr(bridge_name)
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/bridge_lib.py", line 63, in addbr
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent bridge._brctl(['addbr', bridge.name])
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/bridge_lib.py", line 58, in _brctl
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent return ip_wrapper.netns.execute(cmd, run_as_root=True)
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 878, in execute
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent log_fail_as_error=log_fail_as_error, **kwargs)
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent raise RuntimeError(msg)
2016-04-07 17:58:23.406 1 ERROR neutron.plugins.ml2.drivers.agent._common_agent RuntimeError: Exit code: 96; Stdin: ; Stdout: ; Stderr: /var/lib/kolla/venv/bin/neutron-rootwrap: Executable not found: brctl (filter match = brctl)

Another issue is that after removing the openvswitch deployment and cleaning up I was not able to remove the bridges created by openvswitch.
I was getting the error "RTNETLINK answers: Operation not supported"
The only way to remove them was through the openvswitch container using the following commands:

docker exec openvswitch_vswitchd ovs-dpctl del-dp ovs-system
docker exec openvswitch_vswitchd ovs-vsctl del-br br-int
docker exec openvswitch_vswitchd ovs-vsctl del-br br-tun
docker exec openvswitch_vswitchd ovs-vsctl del-br br-ex

Maybe this should added to the cleanup scripts before removing the containers?

[Ahmad Al-Shishtawy (alshishtawy)]

Just wanted to add that in the case of openvswitch this is what I get in /var/log/kolla/neutron/neutron-openvswitch-agent.log

2016-04-07 18:55:39.252 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-1ef53c53-dcb3-43e8-aced-85618be4539c - - - - -] Mapping physical network physnet1 to bridge br-ex
2016-04-07 18:55:40.835 1 ERROR neutron.agent.ovsdb.impl_vsctl [req-1ef53c53-dcb3-43e8-aced-85618be4539c - - - - -] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns$
ofport', 'list', 'Interface', 'int-br-ex']. Exception: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "int-br-ex" in table Interface
..
..
..
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib [req-1ef53c53-dcb3-43e8-aced-85618be4539c - - - - -] Timed out retrieving ofport on port int-br-ex.
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib Traceback (most recent call last):
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 293, in get_port_ofport
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib ofport = self._get_port_ofport(port_name)
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 92, in wrapped
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib return new_fn(*args, **kwargs)
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib return Retrying(*dargs, **dkw).call(f, *args, **kw)
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 214, in call
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib raise RetryError(attempt)
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib RetryError: RetryError[Attempts: 13, Value: None]
2016-04-07 18:55:51.104 1 ERROR neutron.agent.common.ovs_lib
2016-04-07 18:55:51.858 1 ERROR neutron.agent.ovsdb.impl_vsctl [req-1ef53c53-dcb3-43e8-aced-85618be4539c - - - - -] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=
ofport', 'list', 'Interface', 'phy-br-ex']. Exception: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "phy-br-ex" in table Interface
...
...
...
2016-04-07 18:56:02.017 1 ERROR neutron.agent.common.ovs_lib [req-1ef53c53-dcb3-43e8-aced-85618be4539c - - - - -] Timed out retrieving ofport on port phy-br-ex.
2016-04-07 18:56:02.017 1 ERROR neutron.agent.common.ovs_lib Traceback (most recent call last):
2016-04-07 18:56:02.017 1 ERROR neutron.agent.common.ovs_lib File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 293, in get_port_ofport
2016-04-07 18:56:02.017 1 ERROR neutron.agent.common.ovs_lib ofport = self._get_port_ofport(port_name)
2016-04-07 18:56:02.017 1 ERROR neutron.agent.common.ovs_lib File "/var/lib/kolla/venv/lib/python2...

[Steven Dake (sdake)]

Thanks for the bug report. This is actually already actively being worked on. Please try this and add a +1 if it worked for you :)
https://review.openstack.org/#/c/302721/