Contrary to what I said in IRC this morning, I'm actually not on Trusty:

root@undercloud-undercloud-ojtyffepm45g:~# lsb_release -rc
Release:        13.10
Codename:       saucy
root@undercloud-undercloud-ojtyffepm45g:~# ovsdb-server --version
ovsdb-server (Open vSwitch) 1.10.2
Compiled Sep 23 2013 15:02:24
root@undercloud-undercloud-ojtyffepm45g:~# neutron --version
2.3.4.36


I don't have any logs showing problems killing the client; in fact, my /var/log/auth.log shows the kill happening quite successfully:

root@undercloud-undercloud-ojtyffepm45g:/var/log# grep kill auth.log.1 
Mar 24 01:38:53 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf kill -9 20611
root@undercloud-undercloud-ojtyffepm45g:/var/log# zgrep kill auth.log.2.gz 
Mar 20 01:03:19 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf kill -HUP 3891
<<snip>>

I can reproduce the problem through a simple ``service openvswitch-switch restart``; here are the logs I see when I do that:

==> upstart/openvswitch-switch.log <==
 * Killing ovs-vswitchd (1236)
 * Killing ovsdb-server (1226)

==> auth.log <==
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root

==> upstart/neutron-openvswitch-agent.log <==
2014-04-01 22:18:45.198 27450 ERROR neutron.agent.linux.ovsdb_monitor [req-642d9e73-e9fd-4e37-9364-0cc9f89956f6 None] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
openvswitch-switch stop/waiting

==> auth.log <==
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-vsctl --timeout=10 list-ports br-int
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

==> syslog <==
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g ovs-vsctl: 00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g ovs-vsctl: 00002|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

==> auth.log <==
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root

==> upstart/neutron-openvswitch-agent.log <==
2014-04-01 22:18:45.497 27450 ERROR neutron.agent.linux.ovs_lib [req-642d9e73-e9fd-4e37-9364-0cc9f89956f6 None] Unable to execute ['ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']. Exception: 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']
Exit code: 1
Stdout: ''
Stderr: '2014-04-01T22:18:45Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n'
2014-04-01 22:18:45.511 27450 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-642d9e73-e9fd-4e37-9364-0cc9f89956f6 None] Error while processing VIF ports
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1216, in rpc_loop
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_info = self.scan_ports(ports, updated_ports_copy)
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 818, in scan_ports
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     cur_ports = self.int_br.get_vif_port_set()
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 359, in get_vif_port_set
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_names = self.get_port_name_list()
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 315, in get_port_name_list
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     res = self.run_vsctl(["list-ports", self.br_name], check_error=True)
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 73, in run_vsctl
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ctxt.reraise = False
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 68, in __exit__
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 66, in run_vsctl
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return utils.execute(full_args, root_helper=self.root_helper)
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/venvs/neutron/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'list-ports', 'br-int']
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '2014-04-01T22:18:45Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such[210/483]
directory)\n'
2014-04-01 22:18:45.511 27450 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 

==> syslog <==
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=6.12.0

==> upstart/openvswitch-switch.log <==
 * Starting ovsdb-server

==> syslog <==
Apr  1 22:18:45 undercloud-undercloud-ojtyffepm45g ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=1.10.2 "external-ids:system-id=\"cb5c322e-e603-49ed-a397-edd1f75e0467\"" "system-type=\"Ubuntu\"" "system-version=\"13.10-saucy\""

==> upstart/openvswitch-switch.log <==
 * Configuring Open vSwitch system IDs
 * Starting ovs-vswitchd
 * Enabling remote OVSDB managers
openvswitch-switch start/running
root@undercloud-undercloud-ojtyffepm45g:/var/log# 
==> auth.log <==
Apr  1 22:18:47 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-vsctl --timeout=10 list-ports br-int
Apr  1 22:18:47 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr  1 22:18:47 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root
<<snip lots of ovs-vsctl commands>>

To fix it, I can simply restart the neutron-openvswitch-agent service. Logs:

==> syslog <==
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g ovsdb-client: 00001|fatal_signal|WARN|terminating with signal 15 (Terminated)

==> upstart/neutron-openvswitch-agent.log <==
2014-04-01 22:23:39.591 27450 ERROR neutron.agent.linux.ovsdb_monitor [req-642d9e73-e9fd-4e37-9364-0cc9f89956f6 None] Error received from ovsdb monitor: 2014-04-01T22:23:39Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)

==> auth.log <==
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root

==> upstart/neutron-openvswitch-agent.log <==
neutron-openvswitch-agent stop/waiting

==> auth.log <==
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-vsctl --timeout=10 -- --if-exists del-port br-int patch-tun
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

==> syslog <==
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=10 -- --if-exists del-port br-int patch-tun

==> auth.log <==
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl del-flows br-int
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl add-flow br-int hard_timeout=0,idle_timeout=0,priority=1,actions=normal
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl del-flows br-ctlplane
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl add-flow br-ctlplane hard_timeout=0,idle_timeout=0,priority=1,actions=normal
Apr  1 22:23:39 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr  1 22:23:40 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session closed for user root
Apr  1 22:23:40 undercloud-undercloud-ojtyffepm45g sudo:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovs-vsctl --timeout=10 -- --if-exists del-port br-int int-br-ctlplane
Apr  1 22:23:40 undercloud-undercloud-ojtyffepm45g sudo: pam_unix(sudo:session): session opened for user root by (uid=0)



ekarlson noted on irc that after ovsdb-server is restarted. I can confirm that this is the case for me too.

After ``service openvswitch-switch restart``, I waited until neutron-openvswitch-agent had restarted its ovsdb-client. At this point:

root@undercloud-undercloud-ojtyffepm45g:/var/log# ovs-ofctl dump-flows br-int                                                                                                                                                                                                                                                  
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=3.267s, table=0, n_packets=2, n_bytes=284, idle_age=3, priority=0 actions=NORMAL
root@undercloud-undercloud-ojtyffepm45g:/var/log# 

After restarting neutron-openvswitch-agent:

root@undercloud-undercloud-ojtyffepm45g:/var/log# ovs-ofctl dump-flows br-int                                                                                                                                                                                                                                                  
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=4.131s, table=0, n_packets=4, n_bytes=294, idle_age=3, priority=2,in_port=1 actions=drop
 cookie=0x0, duration=3.554s, table=0, n_packets=5, n_bytes=340, idle_age=0, priority=3,in_port=1,vlan_tci=0x0000 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=4.613s, table=0, n_packets=0, n_bytes=0, idle_age=4, priority=1 actions=NORMAL
root@undercloud-undercloud-ojtyffepm45g:/var/log# 


It seems as though neutron-openvswitch-agent is detecting the lack of rules when it restarts; it seems as though it would be helpful if it could run through the same logic after ovsdb-server is restarted