neutron_openvswitch_agent failed to start due to br-ex doesn't exist

[Yongfeng Du]

the error message from openvswitch_agent:

2016-01-14 02:58:54.404 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-5653325b-c948-4c9c-bbce-1b469af7fa27 - - - - -] Mapping physical network physnet1 to bridge br-ex
2016-01-14 02:58:54.405 1 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-5653325b-c948-4c9c-bbce-1b469af7fa27 - - - - -] Bridge br-ex for physical network physnet1 does not exist. Agent terminated!

[Yongfeng Du (dolpherdu)]

The openvswitch_vswitchd has nothing wrong reported:

INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/openvswitch/ovs-vswitchd.log'

And the openvswitch_db is started correctly:
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --log-file=/var/log/openvswitch/ovsdb-server.log'

[Yongfeng Du (dolpherdu)]

br-ex is created by this task ansible/roles/neutron/tasks/start.yml:

...
- name: Ensuring OVS bridge is properly setup
  command: docker exec openvswitch_db /usr/local/bin/kolla_ensure_openvswitch_configured {{ neutron_bridge_name }} {{ neutron_external_interface }}
  register: status
  changed_when: status.stdout.find('changed') != -1
  when:
    - (inventory_hostname in groups['compute'] or inventory_hostname in groups['neutron-agents'])
    - neutron_plugin_agent == "openvswitch"
...

And in the kolla_ensure_openvswitch_configured, br-ex will be created:
    ovs-vsctl --no-wait add-br $bridge

This run after openvswitch_db but before openvswitch_vswitchd.

I think the ovs-vsctl should run after openvswitch_vswitchd and proposed a fix.

That fixed the problem but Sam points out there should be no dependency, so root cause is still unknown.

[Hui Kang (huikang27)]

Hi, Yongfeng, I was trying to reproduce the error.

1. Are you using All-in-one inventory file?
2. if not, did the error happen on network node or compute node?

- Hui

[Yongfeng Du (dolpherdu)]

Hui,

I'm using all in one, running ubuntu install from source.

I did further tests on my env, I can add-br without openvswitch_vswitchd running :
1. stop openvswitch_vswitchd
2. run "ovs-vsctl --no-wait add-br aaa" in the openvswitch_db container
3. docker start openvswitch_vswitchd
4. ovs-vsctl show -- "aaa" is created as expected.

There is still no clue why that failed during "kolla_ansible deploy"

[Xu Ao (xuao)]

I met this problem while deploying openstack by steps .
I just add a br-ex bridge to the device which connected to the internet, and
problem solved.

[Yongfeng Du (dolpherdu)]

Workaround here:
https://review.openstack.org/#/c/266703/

[Jeffrey Zhang (jeffrey4l)]

the root cause is that when run "Ensuring OVS bridge is properly setup" task, the openvswitch_db is not ready. Test in my local with verbose std output:

TASK: [neutron | Ensuring OVS bridge is properly setup] ***********************
<localhost> REMOTE_MODULE command docker exec openvswitch_db /usr/local/bin/kolla_ensure_openvswitch_configured br-ex eth1
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879 && echo $HOME/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879']
<localhost> PUT /tmp/tmpMvPWW9 TO /root/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879/command; rm -rf /root/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879/ >/dev/null 2>&1']
changed: [localhost] => {"changed": true, "cmd": ["docker", "exec", "openvswitch_db", "/usr/local/bin/kolla_ensure_openvswitch_configured", "br-ex", "eth1"], "delta": "0:00:00.224143", "end": "2016-02-03 03:42:42.155264", "rc": 0, "start": "2016-02-03 03:42:41.931121", "stderr": "ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)", "stdout": "changed", "stdout_lines": ["changed"], "warnings": []}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275573
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=a692718c00e0aeb431a7ea932f9a33ffbede0eae
Submitter: Jenkins
Branch: master

commit a692718c00e0aeb431a7ea932f9a33ffbede0eae
Author: Jeffrey Zhang <[e-mail]>
Date: Wed Feb 3 17:22:04 2016 +0800

    Ensuring the openvswitch_db is ready before creating bridges

    Change-Id: I42550c57222de4ad5556e95d40bb96528bc1d085
    Closes-Bug: #1533942

[OpenStack Infra (hudson-openstack)]

Change abandoned by Yongfeng Du (<email address hidden>) on branch: master
Review: https://review.openstack.org/266703
Reason: Fixed by https://review.openstack.org/#/c/275573/