I've been able to track down what I believe is the root problem.
If ovsdb-server (run by the openvswitch-switch service) restarts, the neutron-openvswitch-agent loses its connection and needs to be manually restarted in order to reconnect.
Causes of this bug I've seen have included ovsdb-server segfaulting, being kill -9ed, and being gracefully restarted with "service openvswitch-switch restart".
The errors recorded in /var/log/upstart/neutron-openvswitch-agent.log vary depending on why ovsdb-server went away:
2014-03-23 20:10:01.883 20375 ERROR neutron.agent.linux.ovsdb_monitor [req-a776b981-b86b-4437-ab65-0c6be6070094 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
2014-03-24 01:40:17.617 20375 ERROR neutron.agent.linux.ovsdb_monitor [req-a776b981-b86b-4437-ab65-0c6be6070094 None] Error received from ovsdb monitor: 2014-03-24T01:40:17Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
2014-03-24 04:08:59.718 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
2014-03-24 22:44:22.174 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
2014-03-24 22:44:52.220 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/var/run/openvswitch/db.sock" (Connection refused)
2014-03-24 22:45:22.266 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/var/run/openvswitch/db.sock" (Connection refused)
2014-03-24 22:45:52.310 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/var/run/openvswitch/db.sock" (Connection refused)
2014-03-24 22:46:22.355 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/var/run/openvswitch/db.sock" (Connection refused)
2014-03-24 22:49:27.179 8455 ERROR neutron.agent.linux.ovsdb_monitor [req-d2c2cbd5-a77a-4455-84ac-0a8ec69b41e8 None] Error received from ovsdb monitor: 2014-03-24T22:49:27Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
2014-03-24 22:55:45.441 16033 ERROR neutron.agent.linux.ovsdb_monitor [req-5fe682ce-138e-46d6-aa7e-f0d43ab576ee None] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
In all cases, the result is the same: until neutron-openvswitch-agent is restarted, no traffic is passed onto the tapXXXXX interface inside the dhcp-XXXXX netns
I've been able to track down what I believe is the root problem.
If ovsdb-server (run by the openvswitch-switch service) restarts, the neutron- openvswitch- agent loses its connection and needs to be manually restarted in order to reconnect.
Causes of this bug I've seen have included ovsdb-server segfaulting, being kill -9ed, and being gracefully restarted with "service openvswitch-switch restart".
The errors recorded in /var/log/ upstart/ neutron- openvswitch- agent.log vary depending on why ovsdb-server went away:
2014-03-23 20:10:01.883 20375 ERROR neutron. agent.linux. ovsdb_monitor [req-a776b981- b86b-4437- ab65-0c6be60700 94 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/ run/openvswitch /db.sock: receive failed (End of file) agent.linux. ovsdb_monitor [req-a776b981- b86b-4437- ab65-0c6be60700 94 None] Error received from ovsdb monitor: 2014-03- 24T01:40: 17Z|00001| fatal_signal| WARN|terminatin g with signal 15 (Terminated) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/ run/openvswitch /db.sock: receive failed (End of file) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/ run/openvswitch /db.sock: receive failed (End of file) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/ var/run/ openvswitch/ db.sock" (Connection refused) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/ var/run/ openvswitch/ db.sock" (Connection refused) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/ var/run/ openvswitch/ db.sock" (Connection refused) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: ovsdb-client: failed to connect to "unix:/ var/run/ openvswitch/ db.sock" (Connection refused) agent.linux. ovsdb_monitor [req-d2c2cbd5- a77a-4455- 84ac-0a8ec69b41 e8 None] Error received from ovsdb monitor: 2014-03- 24T22:49: 27Z|00001| fatal_signal| WARN|terminatin g with signal 15 (Terminated) agent.linux. ovsdb_monitor [req-5fe682ce- 138e-46d6- aa7e-f0d43ab576 ee None] Error received from ovsdb monitor: ovsdb-client: unix:/var/ run/openvswitch /db.sock: receive failed (End of file)
2014-03-24 01:40:17.617 20375 ERROR neutron.
2014-03-24 04:08:59.718 8455 ERROR neutron.
2014-03-24 22:44:22.174 8455 ERROR neutron.
2014-03-24 22:44:52.220 8455 ERROR neutron.
2014-03-24 22:45:22.266 8455 ERROR neutron.
2014-03-24 22:45:52.310 8455 ERROR neutron.
2014-03-24 22:46:22.355 8455 ERROR neutron.
2014-03-24 22:49:27.179 8455 ERROR neutron.
2014-03-24 22:55:45.441 16033 ERROR neutron.
In all cases, the result is the same: until neutron- openvswitch- agent is restarted, no traffic is passed onto the tapXXXXX interface inside the dhcp-XXXXX netns