the neutron_agents container failed now and then in the gate

Bug #1541266 reported by Jeffrey Zhang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Confirmed
Undecided
Jeffrey Zhang

Bug Description

check this log[0],

2016-02-02 12:25:38.863 | + failed_containers=neutron_openvswitch_agent
2016-02-02 12:25:38.863 | + for failed in '${failed_containers}'
2016-02-02 12:25:38.863 | + docker logs --tail all neutron_openvswitch_agent
2016-02-02 12:25:38.910 | INFO:__main__:Kolla config strategy set to: COPY_ONCE
2016-02-02 12:25:38.910 | INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
2016-02-02 12:25:38.910 | INFO:__main__:Validating config file
2016-02-02 12:25:38.910 | INFO:__main__:Copying service configuration files
2016-02-02 12:25:38.910 | INFO:__main__:Copying /var/lib/kolla/config_files/neutron.conf to /etc/neutron/neutron.conf
2016-02-02 12:25:38.910 | INFO:__main__:Setting permissions for /etc/neutron/neutron.conf
2016-02-02 12:25:38.911 | INFO:__main__:Copying /var/lib/kolla/config_files/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini
2016-02-02 12:25:38.911 | INFO:__main__:Setting permissions for /etc/neutron/plugins/ml2/ml2_conf.ini
2016-02-02 12:25:38.911 | INFO:__main__:Writing out command to execute
2016-02-02 12:25:38.911 | Running command: 'neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
2016-02-02 12:25:38.911 | No handlers could be found for logger "oslo_reports.guru_meditation_report"
2016-02-02 12:25:38.911 | 2016-02-02 12:24:34.767 1 INFO neutron.common.config [-] Logging enabled!
2016-02-02 12:25:38.911 | 2016-02-02 12:24:34.767 1 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-openvswitch-agent version 8.0.0.0b3.dev117
2016-02-02 12:25:38.912 | 2016-02-02 12:24:37.635 1 INFO neutron.agent.l2.extensions.manager [req-bf7f7e16-13a1-46f6-8dce-3b25e103ccb5 - - - - -] Loaded agent extensions: []
2016-02-02 12:25:38.912 | 2016-02-02 12:24:37.815 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-bf7f7e16-13a1-46f6-8dce-3b25e103ccb5 - - - - -] Mapping physical network physnet1 to bridge br-ex
2016-02-02 12:25:38.912 | 2016-02-02 12:24:37.817 1 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-bf7f7e16-13a1-46f6-8dce-3b25e103ccb5 - - - - -] Bridge br-ex for physical network physnet1 does not exist. Agent terminated!
2016-02-02 12:25:38.915 | + [[ -n neutron_openvswitch_agent ]]

the root cause is that when run "Ensuring OVS bridge is properly setup" task, the openvswitch_db is not ready. Test in my local with verbose std output:

TASK: [neutron | Ensuring OVS bridge is properly setup] ***********************
<localhost> REMOTE_MODULE command docker exec openvswitch_db /usr/local/bin/kolla_ensure_openvswitch_configured br-ex eth1
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879 && echo $HOME/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879']
<localhost> PUT /tmp/tmpMvPWW9 TO /root/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879/command; rm -rf /root/.ansible/tmp/ansible-tmp-1454488961.85-178457803469879/ >/dev/null 2>&1']
changed: [localhost] => {"changed": true, "cmd": ["docker", "exec", "openvswitch_db", "/usr/local/bin/kolla_ensure_openvswitch_configured", "br-ex", "eth1"], "delta": "0:00:00.224143", "end": "2016-02-03 03:42:42.155264", "rc": 0, "start": "2016-02-03 03:42:41.931121", "stderr": "ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)", "stdout": "changed", "stdout_lines": ["changed"], "warnings": []}

[0] http://logs.openstack.org/67/275067/2/check/gate-kolla-dsvm-deploy-ubuntu-source/196e947/console.html#_2016-02-02_12_25_38_859

Changed in kolla:
status: New → Confirmed
assignee: nobody → Jeffrey Zhang (jeffrey4l)
milestone: none → mitaka-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.