neutron

Migration causes downtime while doing bulk_pull

Bug #1801104 reported by Paul Peereboom
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Invalid
High
Unassigned

Bug Description

I'm trying to figure out why some live-migrations in our environment causes unreachability of the instance for about ~3 minutes.

When setting neutron-openvswitch in debug mode we can see that the bulk_pull of resources takes about 2-3 minutes before retrieving about 7057 ports(!). While doing this query the instance is not reachable.

After the instance is migrated to the compute and is pingable again after 3 minutes we can successfully migrate the instance back and forth without downtime on the compute node. It looks like the cache is already filled the second migration and a bulk_pull is not required again. After some time it looks like the cache needs to be refreshed and a live-migration causes the instance to be unreachable for minutes again. We can clear the cache aswell when doing a restart of neutron-openvswitch-agent.

I was able to reproduce this by adding a security group rule with any protocol with a remote-group defined and attach that sec group to an instance. Than migrate the instance to a compute node where neutron-openvswitch is restarted and no other instance is running on it.

Info and logs:
Openstack version: Pike
OS: Redhat

instanceid: 69fd51c3-d5a6-4ce3-94a0-0e66d62634b6
port_id: eb9288fe-9c78-447c-ba77-e2cfd8df624b

Default sec group id: 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 #This sec group causes downtime on migration
Non-default sec group id: 13f878ef-eba7-43fa-9278-19d599bf8f38 #This sec group works as expected

[root@os-admin (openstack_admin)]# openstack security group show 08d7e5c8-03aa-4870-86b0-af5b2db00ac3
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2017-10-11T13:13:48Z |
| description | Default security group |
| id | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| name | default |
| project_id | b9bd8eac1587429aa07ae00661f2a07b |
| revision_number | 16 | |
| updated_at | 2018-04-17T08:14:55Z |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[root@os-admin (openstack_admin)]# openstack security group show 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2018-11-01T12:25:10Z |
| description | |
| id | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 |
| location | None |
| name | temp_sec |
| project_id | b9bd8eac1587429aa07ae00661f2a07b |
| revision_number | 10 | |
| tags | [] |
| updated_at | 2018-11-01T13:22:21Z |
+-----------------+-----------------------------------------------------+

#Default security group, Broken(downtime)
[root@os-admin(openstack_admin)]# openstack security group rule list 08d7e5c8-03aa-4870-86b0-af5b2db00ac3
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 366a55ae-5086-4baf-a55c-bfef0cc38081 | None | None | | None |
| 3e0267fc-3da2-4b9f-a3a0-5facaa1d0819 | tcp | 192.168.99.220/29 | 8000:8000 | None |
| 75ee9c87-1969-421e-bf71-c9244ff1fb49 | icmp | 0.0.0.0/0 | | None |
| b54aaa42-a94a-4e72-9ccd-df8fa691ffb9 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| bffaca24-1a9a-4404-8c21-4ab9d1ed6f46 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| e8cf4dd5-7489-46c4-92f1-970895914f3d | tcp | 192.168.99.220/29 | 22:22 | None |
| fdfad4d8-95bf-419a-aa17-40ad048da7bf | None | None | | None |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+

#temp_sec security group, Works(No downtime)
[root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None |
| 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None |
| 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None |
| 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None |
| f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+

When I migrate an instance with default sec group attached to a compute it's unavailable untill all the ports are retrieved with bulk query (timestamp reachable again: 13:00:03.783):

#########I've added a log rule for the filter and resource type and query_id#########
2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG1: filter: {'security_group_id': (u'08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69
2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG2: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
-packages/neutron/agent/resource_cache.py:70
2018-11-01 12:57:51.951 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG3: query_id: set([('SecurityGroupRule', ('security_group_id', (u'08d7e5c8-03aa-48
70-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

<snip>

2018-11-01 12:58:51.445 291758 DEBUG neutron.agent.resource_cache [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2018-11-01T11:58:50Z,data_plane_status=<?>,description='',device_id='',device_owner='',dhcp_options=
[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=1edbf975-96a1-460f-9576-0bb692615f87,mac_address=fa:16:3e:95:98:90,name='lb-vip-cluster_sub.dev.nl',network_id
=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=3,security=PortSecurity(1edbf975-96a1-460f-9576-0bb692615f87),security_group_ids=set(
[2dd84141-5558-44d8-a2f4-05d998b2c413]),status='DOWN',updated_at=2018-11-01T11:58:50Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

2018-11-01 12:58:51.446 291758 DEBUG neutron_lib.callbacks.manager [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 12:58:51.446 291758 INFO neutron.agent.securitygroups_rpc [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'2dd84141-5558-44d8-a2f4-05d998b2c413'])

2018-11-01 12:58:51.447 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] port_update message processed for port 1edbf975-96a1-460f-9576-0bb692615f87 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 12:58:54.767 291758 DEBUG neutron.agent.resource_cache [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:19Z,data_plane_status=<?>,description='',device_i
d='e4f76e1c-ddff-414a-a108-fdef9a040efd',device_owner='compute:gn2a',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=f46d5036-50a4-4742-9808-326cffb09272,mac_address=fa:16:3e
:08:29:1f,name='lb-node-02.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_po
licy_id=None,revision_number=8,security=PortSecurity(f46d5036-50a4-4742-9808-326cffb09272),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

2018-11-01 12:58:54.768 291758 DEBUG neutron_lib.callbacks.manager [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 12:58:54.768 291758 INFO neutron.agent.securitygroups_rpc [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
-0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])

2018-11-01 12:58:54.769 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd15
1350 - - -] port_update message processed for port f46d5036-50a4-4742-9808-326cffb09272 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 12:58:54.803 291758 DEBUG neutron.agent.resource_cache [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:18Z,data_plane_status=<?>,description='',device_i
d='00e872df-8cd0-4bd4-92a5-f5813ce79a26',device_owner='compute:gn2c',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=b11f828e-21ed-4628-a9f2-2d9e8e2f272b,mac_address=fa:16:3e
:0c:56:9c,name='lb-node-01.patrickdev.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=8,security=PortSecurity(b11f828e-21ed-4628-a9f2-2d9e8e2f272b),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041-ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

2018-11-01 12:58:54.804 291758 DEBUG neutron_lib.callbacks.manager [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 12:58:54.804 291758 INFO neutron.agent.securitygroups_rpc [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
-0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])

2018-11-01 12:58:54.804 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] port_update message processed for port b11f828e-21ed-4628-a9f2-2d9e8e2f272b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 13:00:02.312 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.313 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.315 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

########################
Apparently all our ports in our environment is matched (total of 7057). I've snipped the rest of the resourceid's from the logs
########################

2018-11-01 13:00:03.783 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] 7057 resources returned for queries set([('Port', ('security_group_ids', ('08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

#########################
When I detach the default security group and attach a other security group ( 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 ) to the instance I experience no downtime for the instance while migrating:
#########################

2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: filter: {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69

2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
-packages/neutron/agent/resource_cache.py:70

2018-11-01 12:54:44.130 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: query_id: set([('SecurityGroupRule', ('security_group_id', (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

2018-11-01 12:54:44.130 290907 DEBUG neutron.api.rpc.handlers.resources_rpc [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull called with arguments (<neutron_lib.context.Context object at 0x7f8d06eab3d0>, 'SecurityGroupRule') {'filter_kwargs': {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}} wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66

2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 0fb3d0b8-6b0b-4ae5-93c0-269d6a3f5390 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 1831db02-7538-42ec-b04b-4c8c3d06dadb _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 1c49d1ad-a6f1-470b-bd99-36f32932a6e1 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 3c1d82af-acd7-4543-8922-a96c0c67db67 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 452928bf-d5eb-41ab-b520-f34c33c63058 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 4e9201db-3583-42c7-9c79-372a3394a159 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 4ecc1fa6-d090-4d11-9a3d-323e1f012d40 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 65264a84-ae99-4c90-8626-6dbac869d15d _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: 677d7b02-8943-4824-8fa5-990a02503f03 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: d435b0d1-67bb-4834-9518-5bdf8f843610 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID: fc12d72c-94c9-432b-8f14-79412419c7db _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.165 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] 11 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u
'13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

#########It is returning just 11 resources which is correct value.###############

#########
I can break the working sec group by adding a security group rule with any protocol with a remote-group defined:
#########

[root@os-admin (openstack_admin)]# openstack security group rule create --remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol any --project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1

#Remote group is added:
[root@os-admin (openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None |
| 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None |
| 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None |
| 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None |
| b75497c6-4df2-43cf-b2f9-9a4c04e5e1b3 | None | None | | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 |
| f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+

2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: filter: {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69

2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resourcetype: Port _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:70

2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: query_id: set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

2018-11-01 14:30:24.242 311475 DEBUG neutron.api.rpc.handlers.resources_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull called with arguments (<neutron_lib.context.Context object at 0x7f6dd42d0790>, 'Port') {'filter_kwargs': {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66

2018-11-01 14:30:24.531 311475 DEBUG neutron.agent.resource_cache [req-643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:189

2018-11-01 14:30:24.955 311475 DEBUG neutron.agent.resource_cache [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 826->828). Old fields: {'status': u'DOWN', 'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} New fields: {'status': u'ACTIVE', 'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:24.956 311475 DEBUG neutron_lib.callbacks.manager [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 14:30:24.956 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 14:30:25.124 311475 DEBUG neutron.agent.resource_cache [req-643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:189

2018-11-01 14:30:28.266 311475 DEBUG neutron.agent.resource_cache [req-347c971e-3e22-4805-9b95-9a57d75f98aa a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:189

2018-11-01 14:30:28.380 311475 DEBUG neutron.agent.resource_cache [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 829->830). Old fields: {'status': u'ACTIVE'} New fields: {'status': u'DOWN'} record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:28.381 311475 DEBUG neutron_lib.callbacks.manager [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 14:30:28.381 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 14:30:29.336 311475 DEBUG neutron.agent.resource_cache [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 830->831). Old fields: {'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,profile={"migrating_to": "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-66cc08eb78f8))]} New fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:29.337 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 14:30:29.337 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 14:30:30.007 311475 DEBUG neutron.agent.resource_cache [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number 831->832). Old fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'), 'binding_levels': []} New fields: {'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details={"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": true},vif_type='ovs',vnic_type='normal'), 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute1.odcnoord.nl',level=0,port_id=eb9288fe-9c78-447c-ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:30.008 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 14:30:30.008 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 14:32:37.289 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.290 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 0045bf29-66a6-4277-bfbf-ccc3ff4744e7 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88G
<snip>

#######
7060 resources matched!:
#######
2018-11-01 14:32:38.954 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] 7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security group information for ports [u'eb9288fe-9c78-447c-ba77-e2cfd8df624b'] _apply_port_filter /usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py:135

2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security group information _update_security_group_info /usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py:149

I'm not sure why this behave this way, can someone help me with this issue?

Regards,
Paul Peereboom
ODC-Noord

Paul Peereboom (peereb)
summary: - Migration causes timeout while doing bulk_pull
+ Migration causes downtime while doing bulk_pull
Revision history for this message
zhaobo (zhaobo6) wrote :

So each your operation, the target host didn't host a port, which is associated with your SG.

I pass through the code, seem during process the update port fw filter on agent side "_apply_port_filter" -> "security_group_info_for_devices" -> "SecurityGroupServerAPIShim:security_group_info_for_devices" -> "SecurityGroupInfoAPIMixin:security_group_info_for_ports" -> "_get_security_group_member_ips" "SecurityGroupServerAPIShim:_select_ips_for_remote_group" Then load the whole ports will be processed if the associated SG is defautl SG.

What I confused is the temp_sec(which is new) just contains 11 members. Why query the "7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))])" returned 7060 records? Have you do something during that operation, as just add itselves for remote group "openstack security group rule create --remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol any --project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1:"? Could you please also provide the port info? I found the last log show you are doing the migration, so the log is from the source host or the dest host?

Thank you

tags: added: sg
tags: added: sg-fw
removed: sg
tags: added: pike-backport-potential
Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

What of_interface driver You are using?
Can You check if You have ovsdbapp with patch https://review.openstack.org/#/c/567603/ - maybe that can help to solve Your problem?

Revision history for this message
Paul Peereboom (peereb) wrote :
Download full text (8.7 KiB)

Hi zhaobo,

Thanks for looking in the code. That's correct the target host was empty and didn't host anything. To easily reproduce it needs to be empty and neutron-openvswitch needs to be restarted on the destinaton compute before doing a migration.

>>What I confused is the temp_sec(which is new) just contains 11 members.
Sorry I see I mixed the security groups in the logs, I've seen to much debug logging that day ;-). So I have temp_sec sec group without a remote group id defined and I see that it's not doing a bulk_pull for ports, just for securityGroupRule which returns 5 rules:

2018-11-05 11:23:05.379 398808 DEBUG neutron.api.rpc.handlers.resources_rpc [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method bulk_pull
 called with arguments (<neutron_lib.context.Context object at 0x7fad604f6810>, 'SecurityGroupRule') {'filter_kwargs': {'security_group_id': (u'5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/p
ython2.7/site-packages/oslo_log/helpers.py:66
2018-11-05 11:23:05.405 398808 DEBUG neutron.agent.resource_cache [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] PAUL5: resource ID: 07c52171-065a-4553-b6fb-e43a607bd517 _flood_cache_for_query /usr/
lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-05 11:23:05.405 398808 DEBUG neutron.agent.resource_cache [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] PAUL5: resource ID: 313ab987-6edc-4514-8854-07a5617c2969 _flood_cache_for_query /usr/
lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-05 11:23:05.406 398808 DEBUG neutron.agent.resource_cache [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] PAUL5: resource ID: 374a4917-28a4-478c-b4a0-b3a20eb9979a _flood_cache_for_query /usr/
lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-05 11:23:05.406 398808 DEBUG neutron.agent.resource_cache [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] PAUL5: resource ID: 3d97dd93-e359-4295-bc71-8c3e64f66585 _flood_cache_for_query /usr/
lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-05 11:23:05.406 398808 DEBUG neutron.agent.resource_cache [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] PAUL5: resource ID: f374e5fb-06ec-4aae-b4d2-abc0cae321a0 _flood_cache_for_query /usr/
lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-05 11:23:05.406 398808 DEBUG neutron.agent.resource_cache [req-1d550924-050e-4ff0-8637-0e1a1c647464 - - - - -] 5 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u'5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

Which is correct because that security group has 5 rules:
[root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
 +--------------------------------------+-------------+-------------------+----------...

Read more...

Revision history for this message
Paul Peereboom (peereb) wrote :

>>What of_interface driver You are using?
interface_driver = openvswitch

>>Can You check if You have ovsdbapp with patch https://review.openstack.org/#/c/567603/ - maybe that can help to solve Your problem?

I've patched this on the destination compute but it doesn't change much where does this need to be applied?

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Patch should be applied on destination node so it's not that.
"interface_driver" is something different - it's setting for L3 and DHCP agent. Please check in openvswitch agent's config what You have set in "of_interface" and what is Your "firewall" driver also.

Revision history for this message
Paul Peereboom (peereb) wrote :

Hi Slawek, sorry about that I just grepped in the config for interface_driver.

The of_interface option was deprecated so I already removed it from the config so I think its the default native driver then?

firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Regards,
Paul

Revision history for this message
Paul Peereboom (peereb) wrote :

Update: seems like I'm getting downtime aswell when I restart neutron-openvswitch-agent with an instance on it with a remote group defined. Somehow it does a port bulk_pull and retrieves all ports from my environment and while doing the bulk_pull the instance is not reachable. Anyone have an idea what causes this?

Regards,
Paul

Revision history for this message
Paul Peereboom (peereb) wrote :

After upgrading this site to Queens I cannot reproduce the issue anymore.
Now a migration returns just one resource and there is no timeout when doing a migration:

2018-11-26 10:23:54.312 326318 DEBUG neutron.agent.resource_cache [req-8e2ea784-51f5-49de-ad31-cdcf978d72e3 - - - - -] 1 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b
-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:85

I'll close this issue.
Thanks for your help!

Changed in neutron:
status: Confirmed → Invalid
Revision history for this message
Paul Peereboom (peereb) wrote :

Cannot reproduce anymore after upgrading to Queens.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.