Wish this fix can be fixed into kilo. If it's not able due to the bad timing, wish this fix can be merged into stable/kilo.
-------------------
[The problem]
If there is an Exception (such as DBConnectionError) occurred/occurring when OVS restart,
OVS will return "every thing is OK :-)",
while flow of created network in br-tun will NOT be recovered. :-(
Unless user operation of OVS restart has been executed.
-------------------
[action and log]
[q-agent.log]
[[[create network and subnet and add it to DHCP agent]]]
[[[I turned off MySQL]]]
[[[But nothing happened]]]
[[[Then I restarted OVS]]]
[[[Here it goes...]]]
...
...
...
2015-04-01 22:06:48.237 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Unable to sync tunnel IP 192.168.122.96: Remote error: DBConnectionError (OperationalError) (2003, "Can't connect to MySQL server on '127.0.0.1' (111)") None None
...
...
...
2015-04-01 22:06:56.060 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent tunnel out of sync with plugin!
2015-04-01 22:06:56.061 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is 705639bc86ae44f4b4cc28715ce981e8 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:06:56.062 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is 41f997f166c04cff986ff08eb298b3eb. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:06:56.085 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Unable to sync tunnel IP 192.168.122.96: Remote error: DBConnectionError (OperationalError) (2003, "Can't connect to MySQL server on '127.0.0.1' (111)") None None
...
...
...
2015-04-01 22:06:56.111 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is 6f012243c4844978a7b8181bedcafcc9 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:06:56.112 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is 04ed1ccb78bb4ab495b9ebf40c2338f5. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:06:56.138 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Error while processing VIF ports
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1522, in rpc_loop
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1260, in process_network_ports
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 360, in setup_port_filters
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 219, in decorated_function
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 229, in prepare_devices_filter
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.context, list(device_ids))
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 116, in security_group_info_for_devices
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent devices=devices)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=retry)
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise result
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: DBConnectionError (OperationalError) (2003, "Can't connect to MySQL server on '127.0.0.1' (111)") None None
...
...
...
2015-04-01 22:06:56.138 3698 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
2015-04-01 22:06:56.139 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration:186 completed. Processed ports statistics: {'ancillary': {'removed': 0, 'added': 0}, 'regular': {'updated': 0, 'added': 0, 'removed': 0}}. Elapsed:0.091 loop_count_and_wait /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1411
2015-04-01 22:06:58.049 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration:187 started rpc_loop /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1448
2015-04-01 22:06:58.049 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent out of sync with plugin!
2015-04-01 22:06:58.050 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:06:58.058 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']
Exit code: 0
Stdin:
Stdout: NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=9.952s, table=23, n_packets=0, n_bytes=0, idle_age=9, priority=0 actions=drop
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
...
...
...
------------------------------------------
[[[OK, enough. I'll just start MySQL again...]]]
...
...
...
2015-04-01 22:07:00.192 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['iptables-save', '-c']
Exit code: 0
Stdin:
Stdout: # Generated by iptables-save v1.4.21 on Wed Apr 1 22:07:00 2015
*raw
:PREROUTING ACCEPT [20290:4833947]
:OUTPUT ACCEPT [20523:5745904]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-PREROUTING - [0:0]
[19974:4765842] -A PREROUTING -j neutron-openvswi-PREROUTING
[20207:5677799] -A OUTPUT -j neutron-openvswi-OUTPUT
COMMIT
# Completed on Wed Apr 1 22:07:00 2015
# Generated by iptables-save v1.4.21 on Wed Apr 1 22:07:00 2015
*mangle
...
...
...
COMMIT
# Completed on Wed Apr 1 22:07:00 2015
Stdout:
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.221 DEBUG neutron.agent.linux.iptables_manager [req-352cd26d-7278-483e-a873-7558d0f37acd None None] IPTablesManager.apply completed with success _apply_synchronized /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:468
2015-04-01 22:07:00.222 DEBUG oslo_concurrency.lockutils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Releasing file lock "/opt/stack/data/neutron/lock/neutron-iptables" after holding it for 0.038s release /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-04-01 22:07:00.223 DEBUG oslo_concurrency.lockutils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Releasing semaphore "iptables" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:404
2015-04-01 22:07:00.223 DEBUG neutron.agent.linux.iptables_manager [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Semaphore / lock released "iptables" _apply /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:414
2015-04-01 22:07:00.223 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is 4140d8d1d9b14bd2b2e0e008c12278a4 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.225 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is 4f17d058945744dabfd7e4dffb7c1c85. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.414 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port: 6dd3f67e-ea8c-4c26-868d-a51cfc10581c treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.415 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id=6dd3f67e-ea8c-4c26-868d-a51cfc10581c', 'external_ids:attached-mac!=""'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.420 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', u'external_ids:iface-id=6dd3f67e-ea8c-4c26-868d-a51cfc10581c', 'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[["attached-mac","fa:16:3e:c7:c9:84"],["iface-id","6dd3f67e-ea8c-4c26-868d-a51cfc10581c"],["iface-status","active"]]],"tap6dd3f67e-ea",4]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.421 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', 'tap6dd3f67e-ea'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.427 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', u'tap6dd3f67e-ea']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.427 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port 6dd3f67e-ea8c-4c26-868d-a51cfc10581c updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'dc2e0fc7-b6f3-4602-8c8e-14e1fd9865a9', u'segmentation_id': 1003, u'device_owner': u'network:dhcp', u'physical_network': None, u'mac_address': u'fa:16:3e:c7:c9:84', u'device': u'6dd3f67e-ea8c-4c26-868d-a51cfc10581c', u'port_id': u'6dd3f67e-ea8c-4c26-868d-a51cfc10581c', u'fixed_ips': [{u'subnet_id': u'c6984a53-e9f1-40c2-8504-d166981f1cc6', u'ip_address': u'192.168.50.2'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.428 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', 'tap6dd3f67e-ea'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.434 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', u'tap6dd3f67e-ea']
Exit code: 0
Stdin:
Stdout: {"data":[[1]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.434 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for 6dd3f67e-ea8c-4c26-868d-a51cfc10581c to UP treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.435 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is c1e45d5795b442f8933d3a5ec51e4517 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.435 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is 70ee9752b47b48878c59634d52b77c30. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.505 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device 6dd3f67e-ea8c-4c26-868d-a51cfc10581c completed.
2015-04-01 22:07:00.505 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port: 9d380fa6-81ca-4392-92dc-49016dff05da treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.506 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id=9d380fa6-81ca-4392-92dc-49016dff05da', 'external_ids:attached-mac!=""'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.511 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', u'external_ids:iface-id=9d380fa6-81ca-4392-92dc-49016dff05da', 'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[["attached-mac","fa:16:3e:df:1f:b8"],["iface-id","9d380fa6-81ca-4392-92dc-49016dff05da"],["iface-status","active"]]],"tap9d380fa6-81",2]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.512 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', 'tap9d380fa6-81'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.518 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', u'tap9d380fa6-81']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.519 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port 9d380fa6-81ca-4392-92dc-49016dff05da updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'233087a9-5dd9-4f95-a55e-4e42c2954530', u'segmentation_id': 1001, u'device_owner': u'network:dhcp', u'physical_network': None, u'mac_address': u'fa:16:3e:df:1f:b8', u'device': u'9d380fa6-81ca-4392-92dc-49016dff05da', u'port_id': u'9d380fa6-81ca-4392-92dc-49016dff05da', u'fixed_ips': [{u'subnet_id': u'bfa89ecb-deca-468d-8fcd-b68973dd5337', u'ip_address': u'10.0.0.2'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.520 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', 'tap9d380fa6-81'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.527 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', u'tap9d380fa6-81']
Exit code: 0
Stdin:
Stdout: {"data":[[2]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.527 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for 9d380fa6-81ca-4392-92dc-49016dff05da to UP treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.527 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is 3d08426bf697496fab546a6604e4d802 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.528 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is 9cd3621f640549278d77bed8b4f874cd. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.597 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device 9d380fa6-81ca-4392-92dc-49016dff05da completed.
2015-04-01 22:07:00.597 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port: f677b5e6-2996-49b2-acca-14fe20543cc6 treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.598 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id=f677b5e6-2996-49b2-acca-14fe20543cc6', 'external_ids:attached-mac!=""'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.603 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', u'external_ids:iface-id=f677b5e6-2996-49b2-acca-14fe20543cc6', 'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[["attached-mac","fa:16:3e:62:d4:05"],["iface-id","f677b5e6-2996-49b2-acca-14fe20543cc6"],["iface-status","active"]]],"qr-f677b5e6-29",3]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.603 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', 'qr-f677b5e6-29'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.607 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', u'qr-f677b5e6-29']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.608 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port f677b5e6-2996-49b2-acca-14fe20543cc6 updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'233087a9-5dd9-4f95-a55e-4e42c2954530', u'segmentation_id': 1001, u'device_owner': u'network:router_interface', u'physical_network': None, u'mac_address': u'fa:16:3e:62:d4:05', u'device': u'f677b5e6-2996-49b2-acca-14fe20543cc6', u'port_id': u'f677b5e6-2996-49b2-acca-14fe20543cc6', u'fixed_ips': [{u'subnet_id': u'bfa89ecb-deca-468d-8fcd-b68973dd5337', u'ip_address': u'10.0.0.1'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.608 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', 'qr-f677b5e6-29'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.612 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', u'qr-f677b5e6-29']
Exit code: 0
Stdin:
Stdout: {"data":[[2]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.612 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for f677b5e6-2996-49b2-acca-14fe20543cc6 to UP treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.612 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is 3f3cb1b8657e407da96eb7c5bf26f087 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.613 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is 75c68e8426674f00bb9c2b8078f2c70b. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.670 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device f677b5e6-2996-49b2-acca-14fe20543cc6 completed.
2015-04-01 22:07:00.671 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Processing port: 36c9baf7-a45d-4207-98b5-8155523c7a1b treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1148
2015-04-01 22:07:00.671 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id=36c9baf7-a45d-4207-98b5-8155523c7a1b', 'external_ids:attached-mac!=""'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.677 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', u'external_ids:iface-id=36c9baf7-a45d-4207-98b5-8155523c7a1b', 'external_ids:attached-mac!=""']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[["attached-mac","fa:16:3e:de:6e:c1"],["iface-id","36c9baf7-a45d-4207-98b5-8155523c7a1b"],["iface-status","active"],["vm-uuid","c3a7dfd8-0871-4a7b-9a5f-1b0022ae0bf0"]]],"qvo36c9baf7-a4",5]],"headings":["external_ids","name","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.678 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', 'qvo36c9baf7-a4'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.681 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'iface-to-br', u'qvo36c9baf7-a4']
Exit code: 0
Stdin:
Stdout: br-int
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.682 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Port 36c9baf7-a45d-4207-98b5-8155523c7a1b updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'dc2e0fc7-b6f3-4602-8c8e-14e1fd9865a9', u'segmentation_id': 1003, u'device_owner': u'compute:None', u'physical_network': None, u'mac_address': u'fa:16:3e:de:6e:c1', u'device': u'36c9baf7-a45d-4207-98b5-8155523c7a1b', u'port_id': u'36c9baf7-a45d-4207-98b5-8155523c7a1b', u'fixed_ips': [{u'subnet_id': u'c6984a53-e9f1-40c2-8504-d166981f1cc6', u'ip_address': u'192.168.50.3'}], u'network_type': u'vxlan'}
2015-04-01 22:07:00.682 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', 'qvo36c9baf7-a4'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.686 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=tag', 'list', 'Port', u'qvo36c9baf7-a4']
Exit code: 0
Stdin:
Stdout: {"data":[[1]],"headings":["tag"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.687 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Setting status for 36c9baf7-a45d-4207-98b5-8155523c7a1b to UP treat_devices_added_or_updated /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1175
2015-04-01 22:07:00.687 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is 6e87e064d5b34ef7b293dbcfdfa60340 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.687 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is b6d420f5532d4be9ba30311656b3090e. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.764 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Configuration for device 36c9baf7-a45d-4207-98b5-8155523c7a1b completed.
2015-04-01 22:07:00.765 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] process_network_ports - iteration:188 - treat_devices_added_or_updated completed. Skipped 0 devices of 4 devices currently available. Time elapsed: 0.541 process_network_ports /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1281
2015-04-01 22:07:00.765 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] process_network_ports - iteration:188 - treat_devices_removed completed in 0.000 process_network_ports /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1299
2015-04-01 22:07:00.765 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration:188 - ports processed. Elapsed:0.716 rpc_loop /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1526
2015-04-01 22:07:00.765 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-ports', 'br-ex'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.770 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-ports', u'br-ex']
Exit code: 0
Stdin:
Stdout: qg-66217dea-97
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.771 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--if-exists', '--columns=name,external_ids,ofport', 'list', 'Interface', 'qg-66217dea-97'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:00.777 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--if-exists', '--columns=name,external_ids,ofport', 'list', 'Interface', u'qg-66217dea-97']
Exit code: 0
Stdin:
Stdout: {"data":[["qg-66217dea-97",["map",[["attached-mac","fa:16:3e:6e:00:93"],["iface-id","66217dea-974a-4f2b-bba7-7d4a85cd52b7"],["iface-status","active"]]],1]],"headings":["name","external_ids","ofport"]}
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
2015-04-01 22:07:00.777 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration:188 - ancillary port info retrieved. Elapsed:0.728 rpc_loop /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1542
2015-04-01 22:07:00.778 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is ed1df33c378445579c01e81312539430 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.778 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is b9871a477d554b83ab9229588f9644fa. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.848 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Ancillary Port 66217dea-974a-4f2b-bba7-7d4a85cd52b7 added
2015-04-01 22:07:00.850 DEBUG oslo_messaging._drivers.amqpdriver [req-352cd26d-7278-483e-a873-7558d0f37acd None None] MSG_ID is de4643d4fed4494dbb34e465c3f26bc0 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-04-01 22:07:00.851 DEBUG oslo_messaging._drivers.amqp [req-352cd26d-7278-483e-a873-7558d0f37acd None None] UNIQUE_ID is d4231d08eedc44c8a4e05ff4a3d9ce1a. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-01 22:07:00.913 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] process_ancillary_network_ports - iteration: 188 - treat_ancillary_devices_added completed in 0.136 process_ancillary_network_ports /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1314
2015-04-01 22:07:00.914 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] process_ancillary_network_ports - iteration: 188 - treat_ancillary_devices_removed completed in 0.000 process_ancillary_network_ports /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1330
2015-04-01 22:07:00.914 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration: 188 - ancillary ports processed. Elapsed:0.865 rpc_loop /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1551
2015-04-01 22:07:00.915 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration:188 completed. Processed ports statistics: {'ancillary': {'removed': 0, 'added': 1}, 'regular': {'updated': 0, 'added': 4, 'removed': 0}}. Elapsed:0.865 loop_count_and_wait /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1411
2015-04-01 22:07:02.050 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Agent rpc_loop - iteration:189 started rpc_loop /opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1448
2015-04-01 22:07:02.051 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None] Running command (rootwrap daemon): ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:98
2015-04-01 22:07:02.061 DEBUG neutron.agent.linux.utils [req-352cd26d-7278-483e-a873-7558d0f37acd None None]
Command: ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']
Exit code: 0
Stdin:
Stdout: NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=13.955s, table=23, n_packets=0, n_bytes=0, idle_age=13, priority=0 actions=drop
Stderr: execute /opt/stack/neutron/neutron/agent/linux/utils.py:132
...
...
...
[[[Seems like the log back to peace, again...]]]
[[[But flow of created network does not recovered...]]]
Fix proposed to branch: master
Review: https:/