neutron

Removing gateway ip for tenant network (DVR) causes traceback in neutron-openvswitch-agent

Bug #1728665 reported by James Denton on 2017-10-30

This bug affects 7 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Brian Haley

Bug Description

Version: OpenStack Newton (OSA v14.2.11)
neutron-openvswitch-agent version 9.4.2.dev21

Issue:

Users complained that instances were unable to procure their IP via DHCP. On the controllers, numerous ports were found in BUILD state. Tracebacks similar to the following could be observed in the neutron-openvswitch-agent logs across the (3) controllers.

2017-10-26 16:24:28.458 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port e9c11103-9d10-4b27-b739-e428773d8fac updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'e57257d9-f915-4c60-ac30-76b0e2d36378', u'segmentation_id': 2123, u'device_owner': u'network:dhcp', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:af:aa:f5', u'device': u'e9c11103-9d10-4b27-b739-e428773d8fac', u'port_security_enabled': False, u'port_id': u'e9c11103-9d10-4b27-b739-e428773d8fac', u'fixed_ips': [{u'subnet_id': u'b7196c99-0df6-4b0e-bbfa-e62da96dac86', u'ip_address': u'10.1.1.32'}], u'network_type': u'vlan'}
2017-10-26 16:24:28.458 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Assigning 48 as local vlan for net-id=e57257d9-f915-4c60-ac30-76b0e2d36378
2017-10-26 16:24:28.462 4403 INFO neutron.agent.l2.extensions.qos [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] QoS extension did have no information about the port e9c11103-9d10-4b27-b739-e428773d8fac that we were trying to reset
2017-10-26 16:24:28.462 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 610c3924-5e94-4f95-b19b-75e43c5729ff updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'f09a8be9-a7c7-4f90-8cb3-d08b61095c25', u'segmentation_id': 5, u'device_owner': u'network:router_gateway', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:bf:39:43', u'device': u'610c3924-5e94-4f95-b19b-75e43c5729ff', u'port_security_enabled': False, u'port_id': u'610c3924-5e94-4f95-b19b-75e43c5729ff', u'fixed_ips': [{u'subnet_id': u'3ce21ed4-bb6a-4e67-b222-a055df40af08', u'ip_address': u'96.116.48.132'}], u'network_type': u'vlan'}
2017-10-26 16:24:28.463 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Assigning 43 as local vlan for net-id=f09a8be9-a7c7-4f90-8cb3-d08b61095c25
2017-10-26 16:24:28.466 4403 INFO neutron.agent.l2.extensions.qos [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] QoS extension did have no information about the port 610c3924-5e94-4f95-b19b-75e43c5729ff that we were trying to reset
2017-10-26 16:24:28.467 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 66db7e2d-bd92-48ea-85fa-5e20dfc5311c updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'fd67eae2-9db7-4f7c-a622-39be67090cb4', u'segmentation_id': 2170, u'device_owner': u'network:dhcp', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:c9:24:8a', u'device': u'66db7e2d-bd92-48ea-85fa-5e20dfc5311c', u'port_security_enabled': False, u'port_id': u'66db7e2d-bd92-48ea-85fa-5e20dfc5311c', u'fixed_ips': [{u'subnet_id': u'47366a54-22ca-47a2-b7a0-987257fa83ea', u'ip_address': u'192.168.189.3'}], u'network_type': u'vlan'}
2017-10-26 16:24:28.467 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Assigning 54 as local vlan for net-id=fd67eae2-9db7-4f7c-a622-39be67090cb4
2017-10-26 16:24:28.470 4403 INFO neutron.agent.l2.extensions.qos [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] QoS extension did have no information about the port 66db7e2d-bd92-48ea-85fa-5e20dfc5311c that we were trying to reset
{...snip...}
2017-10-26 16:24:28.501 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port c53c48d4-77a8-4185-bc87-ff999bdfd4a1 updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'06390e9c-6aa4-427a-91dc-5cf2c62be143', u'segmentation_id': 2003, u'device_owner': u'network:router_interface_distributed', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:38:8b:f0', u'device': u'c53c48d4-77a8-4185-bc87-ff999bdfd4a1', u'port_security_enabled': False, u'port_id': u'c53c48d4-77a8-4185-bc87-ff999bdfd4a1', u'fixed_ips': [{u'subnet_id': u'6d20ab59-a8a8-4663-b052-d78fea133c23', u'ip_address': u'192.168.100.1'}], u'network_type': u'vlan'}
2017-10-26 16:24:28.501 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Assigning 9 as local vlan for net-id=06390e9c-6aa4-427a-91dc-5cf2c62be143
2017-10-26 16:24:28.656 4403 INFO neutron.agent.l2.extensions.qos [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] QoS extension did have no information about the port c53c48d4-77a8-4185-bc87-ff999bdfd4a1 that we were trying to reset
2017-10-26 16:24:28.656 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 68e5740d-81e1-4355-bb48-867305f706bc updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'06b4c4da-9de0-447b-8892-ccc45f3393ba', u'segmentation_id': 2122, u'device_owner': u'network:router_interface_distributed', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:2e:33:cc', u'device': u'68e5740d-81e1-4355-bb48-867305f706bc', u'port_security_enabled': False, u'port_id': u'68e5740d-81e1-4355-bb48-867305f706bc', u'fixed_ips': [{u'subnet_id': u'099a6d6e-1353-4b7a-aab3-ede9d592328d', u'ip_address': u'10.10.1.10'}], u'network_type': u'vlan'}
2017-10-26 16:24:28.657 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Assigning 12 as local vlan for net-id=06b4c4da-9de0-447b-8892-ccc45f3393ba
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager [-] ofctl_service: Exception occurred during handler processing. Backtrace from offending handler [_handle_send_msg] servicing event [SendMsgRequest] follows.
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager Traceback (most recent call last):
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/base/app_manager.py", line 290, in _event_loop
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager handler(ev)
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/app/ofctl/service.py", line 141, in _handle_send_msg
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager datapath.send_msg(msg)
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/controller/controller.py", line 334, in send_msg
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager msg.serialize()
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/ofproto/ofproto_parser.py", line 211, in serialize
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager self._serialize_body()
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/ofproto/ofproto_v1_3_parser.py", line 2654, in _serialize_body
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager match_len = self.match.serialize(self.buf, offset)
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/ofproto/ofproto_v1_3_parser.py", line 1008, in serialize
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager field_offset)
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/ryu/ofproto/oxx_fields.py", line 250, in _serialize
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager value_len = len(value)
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager TypeError: object of type 'NoneType' has no len()
2017-10-26 16:24:28.790 4403 ERROR ryu.base.app_manager
2017-10-26 16:24:37.926 4403 INFO neutron.agent.securitygroups_rpc [req-d8721ca3-32dd-401e-bbf9-51b48a068dd3 1388ac7207b9a81e1f106c2ddaae60b0a027ba25e4cc5a2e1c293962319fad62 fb84a528d1464523b9faba201a60bb1d - - -] Security group member updated [u'84f95888-61f2-4299-b7de-1b5bb8854057', u'b069c4af-aa82-410d-8105-5418dbf3fbfe', u'b0e16b93-6276-40a6-ac36-77350e89d349', u'd2a782cb-3dde-4fa4-b6ca-e76023741a46']
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=None,xid=0x42a2e0fb,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10758795138239065597L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'arp_tpa': None, 'eth_type': 2054, 'vlan_vid': 4108}),out_group=0,out_port=0,priority=3,table_id=1) timed out
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Error while processing VIF ports
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2053, in rpc_loop
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, ovs_restarted)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1636, in process_network_ports
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices_added_updated, ovs_restarted))
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1538, in treat_devices_added_or_updated
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1423, in treat_vif_port
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent fixed_ips, device_owner, ovs_restarted)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 799, in port_bound
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent device_owner)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 572, in bind_port_to_dvr
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent device_owner)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 430, in _bind_distributed_router_interface_port
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent vlan_tag=lvm.vlan, gateway_ip=subnet_info['gateway_ip'])
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_dvr_process.py", line 42, in install_dvr_process_ipv4
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent match=match)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 175, in install_drop
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent instructions=[], match=match, **match_kwargs)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 188, in install_instructions
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._send_msg(msg)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/openstack/venvs/neutron-14.2.4/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 89, in _send_msg
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(m)
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: ofctl request version=0x4,msg_type=0xe,msg_len=None,xid=0x42a2e0fb,OFPFlowMod(buffer_id=4294967295,command=0,cookie=10758795138239065597L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'arp_tpa': None, 'eth_type': 2054, 'vlan_vid': 4108}),out_group=0,out_port=0,priority=3,table_id=1) timed out
2017-10-26 16:24:38.791 4403 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-10-26 16:24:38.851 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Agent out of sync with plugin!
2017-10-26 16:24:38.917 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 'tap3f675555-ac' has lost its vlan tag '75'!
2017-10-26 16:24:38.917 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 'tap6616ea67-af' has lost its vlan tag '73'!
2017-10-26 16:24:38.918 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 'qg-610c3924-5e' has lost its vlan tag '43'!
2017-10-26 16:24:38.918 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 'qr-68e5740d-81' has lost its vlan tag '12'!
2017-10-26 16:24:38.918 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 'qr-c53c48d4-77' has lost its vlan tag '9'!
2017-10-26 16:24:38.919 4403 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-00e34b5f-346a-4c33-a71b-822fde6e6f46 - - - - -] Port 'sg-caac51f8-11' has lost its vlan tag '47'!

It turns out that a user removed the gateway IP from their subnet with the following command:

openstack subnet set --gateway none <subnet>

The subnet is attached to a distributed router. The neutron-openvswitch-agent service was restarted at a later time across the controllers, and immediately began logging tracebacks similar to the output provided. The tracebacks occurred on every agent loop, causing many ports on these hosts to not fully processed and remain in a BUILD state. This had a negative impact on other tenants who virtual routers or dhcp servers were not available.

In the lab, setting the gateway ip with 'openstack subnet set --gateway <ip> <subnet>' and restarting the neutron-openvswitch-agent(s) appears to restore functionality.

Tags:

Revision history for this message

James Denton (james-denton) wrote on 2017-10-30:

Download full text (5.2 KiB)

Additional info:

In this output, port 68e5740d-81e1-4355-bb48-867305f706bc is the port formerly associated with the gateway_ip for the respective tenant subnet. The details of which are:

Additional info:

In this output, port 68e5740d-81e1-4355-bb48-867305f706bc is the port formerly associated with the gateway_ip for the respective tenant subnet. The details of which are:

root@controller01-utility-container-8ad9622f:~# neutron port-show 68e5740d-81e1-4355-bb48-867305f706bc
+-----------------------+-----------------------------------------------------------------------------------+
| Field                 | Value                                                                             |
+-----------------------+-----------------------------------------------------------------------------------+
| admin_state_up        | True                                                                              |
| allowed_address_pairs |                                                                                   |
| binding:host_id       |                                                                                   |
| binding:profile       | {}                                                                                |
| binding:vif_details   | {}                                                                                |
| binding:vif_type      | distributed                                                                       |
| binding:vnic_type     | normal                                                                            |
| created_at            | 2017-09-25T19:28:18Z                                                              |
| description           |                                                                                   |
| device_id             | eee05405-f4c7-4a7a-9100-fc09bbfc9d82                                              |
| device_owner          | network:router_interface_distributed                                              |
| extra_dhcp_opts       |                                                                                   |
| fixed_ips             | {"subnet_id": "099a6d6e-1353-4b7a-aab3-ede9d592328d", "ip_address": "10.10.1.10"} |
| id                    | 68e5740d-81e1-4355-bb48-867305f706bc                                              |
| mac_address           | fa:16:3e:2e:33:cc                                                                 |
| name                  |                                                                                   |
| network_id            | 06b4c4da-9de0-447b-8892-ccc45f3393ba                                              |
| port_security_enabled | False                                                                             |
| project_id            | 340c72665c664f52bba3ac61a81fd501                                                  |
| qos_policy_id         |                                                                                   |
| revision_number       | 207200                                                                            |
| security_groups       |                                                                                   |
| status                | ACTIVE                                                                            |
| tenant_id             | 340c72665c664f52bba3ac61a81fd501                                                  |
| updated_at            | 2017-10-26T21:22:41Z                                                              |
+-----------------------+-----------------------------------------------------------------------------------+
root@controller01-utility-container-8ad9622f:~# neutron subnet-show 099a6d6e-1353-4b7a-aab3-ede9d592328d
+-------------------+-----------------------------------------------+
| Field             | Value                                         |
+-------------------+-----------------------------------------------+
| allocation_pools  | {"start": "10.10.2.70", "end": "10.10.2.220"} |
| cidr              | 10.10.0.0/16                                  |
| created_at        | 2017-09-25T19:17:47Z                          |
| description       |                                               |
| dns_nameservers   | x.x.x.x                                       |
|                   | y.y.y.y                                       |
| enable_dhcp       | True                                          |
| gateway_ip        | None                                          |
| host_routes       |                                               |
| id                | 099a6d6e-1353-4b7a-aab3-ede9d592328d          |
| ip_version        | 4                                             |
| ipv6_address_mode |                                               |
| ipv6_ra_mode      |                                               |
| name              | floating-ipv4                                 |
| network_id        | 06b4c4da-9de0-447b-8892-ccc45f3393ba          |
| project_id        | 340c72665c664f52bba3ac61a81fd501              |
| revision_number   | 5                                             |
| service_types     |                                               |
| subnetpool_id     |                                               |
| tenant_id         | 340c72665c664f52bba3ac61a81fd501              |
| updated_at        | 2017-10-26T21:12:05Z                          |
+-------------------+-----------------------------------------------+

Revision history for this message

James Denton (james-denton) wrote on 2017-10-31:

Download full text (33.0 KiB)

I was able to duplicate this in neutron version 11.0.0.0rc2.dev368 (OpenStack-Ansible master) under the same conditions. The following log snippet is from neutron-openvswitch-agent.log:

2017-10-31 14:20:24.288 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=None,xid=0xfda6e61,OFPFlowMod(buffer_id=4294967295,command=0,cookie=1900617720212172640L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'arp_tpa': None, 'eth_type': 2054, 'vlan_vid': 4098}),out_group=0,out_port=0,priority=3,table_id=1) timed out: Timeout: 10 seconds
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Error while processing VIF ports: RuntimeError: ofctl request version=0x4,msg_type=0xe,msg_len=None,xid=0xfda6e61,OFPFlowMod(buffer_id=4294967295,command=0,cookie=1900617720212172640L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'arp_tpa': None, 'eth_type': 2054, 'vlan_vid': 4098}),out_group=0,out_port=0,priority=3,table_id=1) timed out
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2063, in rpc_loop
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1646, in process_network_ports
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, ovs_restarted))
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1548, in treat_devices_added_or_updated
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1431, in treat_vif_port
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     fixed_ips, device_owner, ovs_restarted)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 794, in port_bound
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     device_owner)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 568, in bind_port_to_dvr
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     device_owner)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 426, in _bind_distributed_router_interface_port
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     vlan_tag=lvm.vlan, gateway_ip=subnet_info['gateway_ip'])
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_dvr_process.py", line 42, in install_dvr_process_ipv4
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     match=match)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 193, in install_drop
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     instructions=[], match=match, **match_kwargs)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 214, in install_instructions
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._send_msg(msg)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 95, in _send_msg
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: ofctl request version=0x4,msg_type=0xe,msg_len=None,xid=0xfda6e61,OFPFlowMod(buffer_id=4294967295,command=0,cookie=1900617720212172640L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],match=OFPMatch(oxm_fields={'arp_tpa': None, 'eth_type': 2054, 'vlan_vid': 4098}),out_group=0,out_port=0,priority=3,table_id=1) timed out
2017-10-31 14:20:24.290 10631 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2017-10-31 14:20:24.293 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Agent rpc_loop - iteration:2 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 6, 'removed': 0}}. Elapsed:10.127 loop_count_and_wait /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1796
2017-10-31 14:20:24.294 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Loop iteration exceeded interval (5 vs. 10.1274929047)! loop_count_and_wait /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1803
2017-10-31 14:20:24.295 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Agent rpc_loop - iteration:3 started rpc_loop /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1976
2017-10-31 14:20:24.297 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x4ab14d82,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1) result [OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=12551188769968443967L,duration_nsec=149000000,duration_sec=32,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)] _send_msg /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2017-10-31 14:20:24.300 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Agent rpc_loop - iteration:3 - starting polling. Elapsed:0.005 rpc_loop /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2031
2017-10-31 14:20:24.301 10631 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Agent out of sync with plugin!
2017-10-31 14:20:24.302 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): ListPortsCommand(bridge=br-int) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.302 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.303 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'qr-b5063eaa-2a', u'sg-4f346d4c-eb', u'qg-ac86fc38-ef', u'fg-3af2714c-ee', u'tap3aa023f7-f9', u'int-br-provider', u'tapce4f293c-d4'], table=Interface, columns=['name', 'external_ids', 'ofport'], row=False) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.305 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.306 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): ListPortsCommand(bridge=br-int) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.307 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.308 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'qr-b5063eaa-2a', u'sg-4f346d4c-eb', u'qg-ac86fc38-ef', u'fg-3af2714c-ee', u'tap3aa023f7-f9', u'int-br-provider', u'tapce4f293c-d4'], table=Port, columns=['name', 'tag'], row=False) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.309 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.310 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): ListPortsCommand(bridge=br-int) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.311 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.311 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'qr-b5063eaa-2a', u'sg-4f346d4c-eb', u'qg-ac86fc38-ef', u'fg-3af2714c-ee', u'tap3aa023f7-f9', u'int-br-provider', u'tapce4f293c-d4'], table=Interface, columns=['name', 'external_ids', 'ofport'], row=False) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.313 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.314 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Agent rpc_loop - iteration:3 - port information retrieved. Elapsed:0.019 rpc_loop /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2054
2017-10-31 14:20:24.314 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Starting to process devices in:{'current': set([u'ce4f293c-d486-4973-b9b7-29ad9bee7eee', u'3af2714c-eec6-4dcd-b44b-e2fbb08d8eca', u'b5063eaa-2a11-43f7-957d-a3aa5cce4748', u'ac86fc38-efbe-4ca1-86a4-0297e1f358bb', u'4f346d4c-eb4c-42c3-be21-5a86e7cb7097', u'3aa023f7-f998-4502-b699-dfba9a21045c']), 'removed': set([]), 'added': set([u'ce4f293c-d486-4973-b9b7-29ad9bee7eee', u'3af2714c-eec6-4dcd-b44b-e2fbb08d8eca', u'b5063eaa-2a11-43f7-957d-a3aa5cce4748', u'ac86fc38-efbe-4ca1-86a4-0297e1f358bb', u'4f346d4c-eb4c-42c3-be21-5a86e7cb7097', u'3aa023f7-f998-4502-b699-dfba9a21045c'])} rpc_loop /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2061
2017-10-31 14:20:24.315 10631 DEBUG neutron.agent.rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Returning: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', 'segmentation_id': 201, 'fixed_ips': [{'subnet_id': 'd7a7901f-8452-4c07-be02-6f4cc1521a4c', 'ip_address': '10.10.10.2'}], 'device_owner': u'network:dhcp', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:46:2e:7d', 'device': u'ce4f293c-d486-4973-b9b7-29ad9bee7eee', 'port_security_enabled': False, 'port_id': 'ce4f293c-d486-4973-b9b7-29ad9bee7eee', 'network_type': u'vlan', 'security_groups': []} get_device_details /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/agent/rpc.py:264
2017-10-31 14:20:24.316 10631 DEBUG neutron.agent.rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Returning: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', 'segmentation_id': 201, 'fixed_ips': [{'subnet_id': 'd7a7901f-8452-4c07-be02-6f4cc1521a4c', 'ip_address': '10.10.10.6'}], 'device_owner': u'network:floatingip_agent_gateway', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:93:a8:1a', 'device': u'3af2714c-eec6-4dcd-b44b-e2fbb08d8eca', 'port_security_enabled': False, 'port_id': '3af2714c-eec6-4dcd-b44b-e2fbb08d8eca', 'network_type': u'vlan', 'security_groups': []} get_device_details /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/agent/rpc.py:264
2017-10-31 14:20:24.317 10631 DEBUG neutron.agent.rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Returning: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '93bf45df-63d1-4ef7-b1ab-e67db1ff5b0e', 'segmentation_id': 200, 'fixed_ips': [{'subnet_id': '840940c3-f766-4dc2-b2e8-8d3eb0ceb20f', 'ip_address': '192.168.1.1'}], 'device_owner': u'network:router_interface_distributed', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:7f:d4:b5', 'device': u'b5063eaa-2a11-43f7-957d-a3aa5cce4748', 'port_security_enabled': False, 'port_id': 'b5063eaa-2a11-43f7-957d-a3aa5cce4748', 'network_type': u'vlan', 'security_groups': []} get_device_details /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/agent/rpc.py:264
2017-10-31 14:20:24.318 10631 DEBUG neutron.agent.rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Returning: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', 'segmentation_id': 201, 'fixed_ips': [{'subnet_id': 'd7a7901f-8452-4c07-be02-6f4cc1521a4c', 'ip_address': '10.10.10.8'}], 'device_owner': u'network:router_gateway', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:39:39:e4', 'device': u'ac86fc38-efbe-4ca1-86a4-0297e1f358bb', 'port_security_enabled': False, 'port_id': 'ac86fc38-efbe-4ca1-86a4-0297e1f358bb', 'network_type': u'vlan', 'security_groups': []} get_device_details /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/agent/rpc.py:264
2017-10-31 14:20:24.318 10631 DEBUG neutron.agent.rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Returning: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '93bf45df-63d1-4ef7-b1ab-e67db1ff5b0e', 'segmentation_id': 200, 'fixed_ips': [{'subnet_id': '840940c3-f766-4dc2-b2e8-8d3eb0ceb20f', 'ip_address': '192.168.1.2'}], 'device_owner': u'network:dhcp', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:32:40:3d', 'device': u'3aa023f7-f998-4502-b699-dfba9a21045c', 'port_security_enabled': False, 'port_id': '3aa023f7-f998-4502-b699-dfba9a21045c', 'network_type': u'vlan', 'security_groups': []} get_device_details /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/agent/rpc.py:264
2017-10-31 14:20:24.319 10631 DEBUG neutron.agent.rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Returning: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '93bf45df-63d1-4ef7-b1ab-e67db1ff5b0e', 'segmentation_id': 200, 'fixed_ips': [{'subnet_id': '840940c3-f766-4dc2-b2e8-8d3eb0ceb20f', 'ip_address': '192.168.1.7'}], 'device_owner': u'network:router_centralized_snat', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:af:09:ca', 'device': u'4f346d4c-eb4c-42c3-be21-5a86e7cb7097', 'port_security_enabled': False, 'port_id': '4f346d4c-eb4c-42c3-be21-5a86e7cb7097', 'network_type': u'vlan', 'security_groups': []} get_device_details /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/agent/rpc.py:264
2017-10-31 14:20:24.320 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): ListPortsCommand(bridge=br-int) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.321 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.322 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'qr-b5063eaa-2a', u'sg-4f346d4c-eb', u'qg-ac86fc38-ef', u'fg-3af2714c-ee', u'tap3aa023f7-f9', u'int-br-provider', u'tapce4f293c-d4'], table=Interface, columns=['name', 'external_ids', 'ofport'], row=False) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.323 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.324 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Processing port: ce4f293c-d486-4973-b9b7-29ad9bee7eee treat_devices_added_or_updated /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1527
2017-10-31 14:20:24.324 10631 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Port ce4f293c-d486-4973-b9b7-29ad9bee7eee updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', 'segmentation_id': 201, 'fixed_ips': [{'subnet_id': 'd7a7901f-8452-4c07-be02-6f4cc1521a4c', 'ip_address': '10.10.10.2'}], 'device_owner': u'network:dhcp', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:46:2e:7d', 'device': u'ce4f293c-d486-4973-b9b7-29ad9bee7eee', 'port_security_enabled': False, 'port_id': 'ce4f293c-d486-4973-b9b7-29ad9bee7eee', 'network_type': u'vlan', 'security_groups': []}
2017-10-31 14:20:24.326 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbGetCommand(column=other_config, table=Port, record=tapce4f293c-d4) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.326 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.327 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbSetCommand(table=Port, col_values=(('other_config', {u'net_uuid': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', u'segmentation_id': '201', u'tag': u'1', u'physical_network': 'physnet1', u'network_type': u'vlan'}),), record=tapce4f293c-d4) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.328 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.329 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Processing port: 3af2714c-eec6-4dcd-b44b-e2fbb08d8eca treat_devices_added_or_updated /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1527
2017-10-31 14:20:24.329 10631 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Port 3af2714c-eec6-4dcd-b44b-e2fbb08d8eca updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', 'segmentation_id': 201, 'fixed_ips': [{'subnet_id': 'd7a7901f-8452-4c07-be02-6f4cc1521a4c', 'ip_address': '10.10.10.6'}], 'device_owner': u'network:floatingip_agent_gateway', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:93:a8:1a', 'device': u'3af2714c-eec6-4dcd-b44b-e2fbb08d8eca', 'port_security_enabled': False, 'port_id': '3af2714c-eec6-4dcd-b44b-e2fbb08d8eca', 'network_type': u'vlan', 'security_groups': []}
2017-10-31 14:20:24.330 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbGetCommand(column=other_config, table=Port, record=fg-3af2714c-ee) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.331 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.332 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbSetCommand(table=Port, col_values=(('other_config', {u'net_uuid': '9ce308e0-7232-4e8d-9f87-d8fe309c7964', u'segmentation_id': '201', u'tag': u'1', u'physical_network': 'physnet1', u'network_type': u'vlan'}),), record=fg-3af2714c-ee) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.333 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.333 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Processing port: b5063eaa-2a11-43f7-957d-a3aa5cce4748 treat_devices_added_or_updated /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1527
2017-10-31 14:20:24.334 10631 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] Port b5063eaa-2a11-43f7-957d-a3aa5cce4748 updated. Details: {'profile': {}, 'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id': '93bf45df-63d1-4ef7-b1ab-e67db1ff5b0e', 'segmentation_id': 200, 'fixed_ips': [{'subnet_id': '840940c3-f766-4dc2-b2e8-8d3eb0ceb20f', 'ip_address': '192.168.1.1'}], 'device_owner': u'network:router_interface_distributed', 'physical_network': u'physnet1', 'mac_address': 'fa:16:3e:7f:d4:b5', 'device': u'b5063eaa-2a11-43f7-957d-a3aa5cce4748', 'port_security_enabled': False, 'port_id': 'b5063eaa-2a11-43f7-957d-a3aa5cce4748', 'network_type': u'vlan', 'security_groups': []}
2017-10-31 14:20:24.335 10631 DEBUG neutron.api.rpc.handlers.dvr_rpc [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] neutron.api.rpc.handlers.dvr_rpc.DVRServerRpcApi method get_ports_on_host_by_subnet called with arguments (<neutron_lib.context.ContextBase object at 0x7f309708f790>, 'aio1-neutron-agents-container-ece80330', '840940c3-f766-4dc2-b2e8-8d3eb0ceb20f') {} wrapper /openstack/venvs/neutron-master/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-31 14:20:24.412 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] DVR: List of ports received from get_ports_on_host_by_subnet [{u'status': u'ACTIVE', u'name': u'', u'admin_state_up': True, u'network_id': u'93bf45df-63d1-4ef7-b1ab-e67db1ff5b0e', u'tenant_id': u'50a21f82d79a4c018dd84f2c034260dc', u'device_owner': u'network:dhcp', u'mac_address': u'fa:16:3e:32:40:3d', u'project_id': u'50a21f82d79a4c018dd84f2c034260dc', u'fixed_ips': [{u'subnet_id': u'840940c3-f766-4dc2-b2e8-8d3eb0ceb20f', u'ip_address': u'192.168.1.2'}], u'id': u'3aa023f7-f998-4502-b699-dfba9a21045c', u'device_id': u'dhcpb6c0b2d1-de5b-5471-a468-968cca3050a0-93bf45df-63d1-4ef7-b1ab-e67db1ff5b0e'}] _bind_distributed_router_interface_port /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py:387
2017-10-31 14:20:24.415 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): ListPortsCommand(bridge=br-int) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.416 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.419 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): DbListCommand(if_exists=True, records=[u'qr-b5063eaa-2a', u'sg-4f346d4c-eb', u'qg-ac86fc38-ef', u'fg-3af2714c-ee', u'tap3aa023f7-f9', u'int-br-provider', u'tapce4f293c-d4'], table=Interface, columns=['name', 'external_ids', 'ofport'], row=False) do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2017-10-31 14:20:24.422 10631 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Transaction caused no change do_commit /openstack/venvs/neutron-master/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:110
2017-10-31 14:20:24.425 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x68,xid=0x4ab14d84,OFPFlowMod(buffer_id=4294967295,command=0,cookie=12551188769968443967L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionSetField(eth_src=u'fa:16:3e:7f:d4:b5')],len=24,type=4), OFPInstructionGotoTable(len=8,table_id=60,type=1)],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:32:40:3d', 'vlan_vid': 4296}),out_group=0,out_port=0,priority=4,table_id=2) result None _send_msg /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2017-10-31 14:20:24.427 10631 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-9f88e2a1-069e-4321-865f-e95c97575c45 - - - - -] ofctl request version=0x4,msg_type=0xe,msg_len=0x68,xid=0x4ab14d86,OFPFlowMod(buffer_id=4294967295,command=0,cookie=12551188769968443967L,cookie_mask=0,flags=0,hard_timeout=0,idle_timeout=0,instructions=[OFPInstructionActions(actions=[OFPActionPopVlan(len=8,type=18), OFPActionOutput(len=16,max_len=0,port=4,type=0)],len=32,type=4)],match=OFPMatch(oxm_fields={'eth_dst': 'fa:16:3e:32:40:3d', 'vlan_vid': 4296}),out_group=0,out_port=0,priority=4,table_id=60) result None _send_msg /openstack/venvs/neutron-master/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:99
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager [-] ofctl_service: Exception occurred during handler processing. Backtrace from offending handler [_handle_send_msg] servicing event [SendMsgRequest] follows.: TypeError: object of type 'NoneType' has no len()
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager Traceback (most recent call last):
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/base/app_manager.py", line 290, in _event_loop
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     handler(ev)
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/app/ofctl/service.py", line 141, in _handle_send_msg
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     datapath.send_msg(msg)
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/controller/controller.py", line 347, in send_msg
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     msg.serialize()
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/ofproto/ofproto_parser.py", line 270, in serialize
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     self._serialize_body()
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/ofproto/ofproto_v1_3_parser.py", line 2708, in _serialize_body
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     match_len = self.match.serialize(self.buf, offset)
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/ofproto/ofproto_v1_3_parser.py", line 1029, in serialize
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     field_offset)
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager   File "/openstack/venvs/neutron-master/lib/python2.7/site-packages/ryu/ofproto/oxx_fields.py", line 253, in _serialize
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager     value_len = len(value)
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager TypeError: object of type 'NoneType' has no len()
2017-10-31 14:20:24.429 10631 ERROR ryu.base.app_manager

Revision history for this message

Brian Haley (brian-haley) wrote on 2017-10-31:

The last comment shows a timeout, as well as a ryu traceback. Is there a problem with this agent communicating with OVS?

tags:

added: ovs

Revision history for this message

James Denton (james-denton) wrote on 2017-10-31:

No communication problem that I can see. I can consistently create this condition by setting gateway_ip=None on a tenant subnet attached to a distributed router and restarting the OVS agent. Setting a valid gateway IP and restarting the agent restores service.

Brian Haley (brian-haley) on 2017-11-02

tags:

added: l3-dvr-backlog

Brian Haley (brian-haley) on 2017-11-17

Changed in neutron:
status:	New → Confirmed
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-17: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/521199

Changed in neutron:
assignee:	nobody → Brian Haley (brian-haley)
status:	Confirmed → In Progress

Revision history for this message

Brian Haley (brian-haley) wrote on 2018-02-01:

Changed status to High since the agent isn't usable once it gets in this state.

Changed in neutron:
importance:	Medium → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-20: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/521199
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9be7b62f773d3f61da57c151bfbd5c8fe4d4e863
Submitter: Zuul
Branch: master

commit 9be7b62f773d3f61da57c151bfbd5c8fe4d4e863
Author: Brian Haley <email address hidden>
Date: Fri Nov 17 16:53:41 2017 -0500

DVR: verify subnet has gateway_ip before installing IPv4 flow

    If a user clears the gateway_ip of a subnet and the OVS
    agent is re-started, it will throw an exception trying
    to install the DVR IPv4 flow. Do not install the flow
    in this case since it is not required.

Change-Id: I79aba63498aa9af1156e37530627fcaec853a740
Closes-bug: #1728665

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

Arjun Baindur (abaindur) wrote on 2018-02-23:

Is this going to be backported into pike? It affects releases all the way back to newton. How was it not disocvered until just now? Seems like a fairly common use case, as whenever we have multi NIC VMs attached to more than one network, gateway is usually disabled on those 2ndary networks.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-28: Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/548605

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-28: Fix proposed to neutron (stable/pike)

#10

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/548606

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-28: Fix proposed to neutron (stable/ocata)

#11

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/548607

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-05: Fix merged to neutron (stable/queens)

#12

Reviewed: https://review.openstack.org/548605
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4593069cd1c3bfd157e7bf7d1fde342a04a5104b
Submitter: Zuul
Branch: stable/queens

commit 4593069cd1c3bfd157e7bf7d1fde342a04a5104b
Author: Brian Haley <email address hidden>
Date: Fri Nov 17 16:53:41 2017 -0500

DVR: verify subnet has gateway_ip before installing IPv4 flow

    Change-Id: I79aba63498aa9af1156e37530627fcaec853a740
    Closes-bug: #1728665
    (cherry picked from commit 9be7b62f773d3f61da57c151bfbd5c8fe4d4e863)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-06: Fix merged to neutron (stable/pike)

#13

Reviewed: https://review.openstack.org/548606
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e0c0b8fa7e01c01d90c096b3b36c6a38b55752e
Submitter: Zuul
Branch: stable/pike

commit 8e0c0b8fa7e01c01d90c096b3b36c6a38b55752e
Author: Brian Haley <email address hidden>
Date: Fri Nov 17 16:53:41 2017 -0500

DVR: verify subnet has gateway_ip before installing IPv4 flow

    Change-Id: I79aba63498aa9af1156e37530627fcaec853a740
    Closes-bug: #1728665
    (cherry picked from commit 9be7b62f773d3f61da57c151bfbd5c8fe4d4e863)

tags:

added: in-stable-pike

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-07: Fix merged to neutron (stable/ocata)

#14

Reviewed: https://review.openstack.org/548607
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f5e7420da5af69293276dc31fe28ff843a74a0b0
Submitter: Zuul
Branch: stable/ocata

commit f5e7420da5af69293276dc31fe28ff843a74a0b0
Author: Brian Haley <email address hidden>
Date: Fri Nov 17 16:53:41 2017 -0500

DVR: verify subnet has gateway_ip before installing IPv4 flow

    Change-Id: I79aba63498aa9af1156e37530627fcaec853a740
    Closes-bug: #1728665
    (cherry picked from commit 9be7b62f773d3f61da57c151bfbd5c8fe4d4e863)