This started happening a few times after some concurrency tests were run which resulted in a different issue reported here:
https://bugs.launchpad.net/group-based-policy/+bug/1513014
At some time, the VM launch started failing. Restarting Opflex agent/neutron server did not help.
Error observed on Neutron:
2015-11-04 04:02:06.034 15352 WARNING gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 None] XYZ-END details --- {'subnets': [{'ipv6_ra_mode': None, 'allocation_pools': [{'start': u'122.0.0.2', 'end': u'122.0.0.254'}], 'host_routes': [{'nexthop': u'122.0.0.1', 'destination': '0.0.0.0/0'}, {'nexthop': u'122.0.0.3', 'destination': '169.254.169.254/16'}], 'dhcp_server_ips': [u'122.0.0.3'], 'ipv6_address_mode': None, 'cidr': u'122.0.0.0/24', 'id': u'018b451f-0a06-4adf-b738-85092e0c9e26', 'name': u'apic_owned_L2POLICY.8ef17cae', 'enable_dhcp': True, 'network_id': u'263f3d48-21f3-4c05-8c6c-1523cc1397d7', 'tenant_id': u'e356572fa5a14bed9f97250712903cac', 'dns_nameservers': [u'122.0.0.3'], 'gateway_ip': u'122.0.0.1', 'ip_version': 4L, 'shared': False}], 'enable_metadata_optimization': False, 'allowed_address_pairs': [], 'vrf_tenant': u'_noirolab_services', 'promiscuous_mode': True, 'ip_mapping': [{'nat_epg_name': u'NAT-epg-Datacenter-Out', 'external_segment_name': u'Datacenter-Out', 'nat_epg_tenant': 'common'}], 'floating_ip': [], 'host': u'fab102-compute-1.dmzlab.local', 'l3_policy_id': u'865b9d6c-6b6d-452d-baf5-f122f41902b4', 'vrf_subnets': [u'122.0.0.0/22', u'192.168.0.0/16'], 'device': u'096fb9c3-75b5-47ea-8e34-e18b362f31c7', 'l2_policy_id': u'84e9799f-253e-47f4-82bf-ef15051aba9a', 'enable_dhcp_optimization': True, 'port_id': u'096fb9c3-75b5-47ea-8e34-e18b362f31c7', 'tenant_id': u'e356572fa5a14bed9f97250712903cac', 'app_profile_name': 'noirolab_app', 'vrf_name': 'advanced_services_monitoring_l3policy', 'mac_address': u'fa:16:3e:2d:85:c4', 'extra_ips': [], 'endpoint_group_name': 'Shd-L2POLICY.8ef17cae', 'ptg_tenant': u'_noirolab_services'}
2015-11-04 04:02:06.071 15352 WARNING gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 None] get owned address ============ set([]) ====== u'4a48acd1-268c-4351-949b-06cf2ed390db'
2015-11-04 04:02:06.124 15352 WARNING gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 None] XYZ-1 - Before _is_pt_chain_head. details = {'subnets': [{'ipv6_ra_mode': None, 'allocation_pools': [{'start': u'122.0.0.2', 'end': u'122.0.0.254'}], 'host_routes': [{'nexthop': u'122.0.0.1', 'destination': '0.0.0.0/0'}, {'nexthop': u'122.0.0.3', 'destination': '169.254.169.254/16'}], 'dhcp_server_ips': [u'122.0.0.3'], 'ipv6_address_mode': None, 'cidr': u'122.0.0.0/24', 'id': u'c8573bba-4843-4159-b225-6da3db950c32', 'name': u'apic_owned_L2POLICY.8ef17cae', 'enable_dhcp': True, 'network_id': u'a39da21a-e002-4a22-9416-33a2cabd4686', 'tenant_id': u'e356572fa5a14bed9f97250712903cac', 'dns_nameservers': [u'122.0.0.3'], 'gateway_ip': u'122.0.0.1', 'ip_version': 4L, 'shared': False}], 'enable_metadata_optimization': False, 'allowed_address_pairs': [], 'vrf_tenant': u'_noirolab_services', 'promiscuous_mode': True, 'ip_mapping': [{'nat_epg_name': u'NAT-epg-Datacenter-Out', 'external_segment_name': u'Datacenter-Out', 'nat_epg_tenant': 'common'}], 'floating_ip': [], 'host': u'fab102-compute-1.dmzlab.local', 'l3_policy_id': u'865b9d6c-6b6d-452d-baf5-f122f41902b4', 'vrf_subnets': [u'122.0.0.0/22', u'192.168.0.0/16'], 'device': u'4a48acd1-268c-4351-949b-06cf2ed390db', 'l2_policy_id': u'b2355b5d-4bf7-405b-a752-5492af4da971', 'enable_dhcp_optimization': True, 'port_id': u'4a48acd1-268c-4351-949b-06cf2ed390db', 'tenant_id': u'e356572fa5a14bed9f97250712903cac', 'app_profile_name': 'noirolab_app', 'vrf_name': 'advanced_services_monitoring_l3policy', 'mac_address': u'fa:16:3e:52:0f:fa', 'extra_ips': [], 'endpoint_group_name': 'Shd-L2POLICY.8ef17cae', 'ptg_tenant': u'_noirolab_services'}
2015-11-04 04:02:06.124 15352 WARNING gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 None] VK pt = None , ptg = None
2015-11-04 04:02:06.124 15352 WARNING gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 None] XYZ-END details --- {'subnets': [{'ipv6_ra_mode': None, 'allocation_pools': [{'start': u'122.0.0.2', 'end': u'122.0.0.254'}], 'host_routes': [{'nexthop': u'122.0.0.1', 'destination': '0.0.0.0/0'}, {'nexthop': u'122.0.0.3', 'destination': '169.254.169.254/16'}], 'dhcp_server_ips': [u'122.0.0.3'], 'ipv6_address_mode': None, 'cidr': u'122.0.0.0/24', 'id': u'c8573bba-4843-4159-b225-6da3db950c32', 'name': u'apic_owned_L2POLICY.8ef17cae', 'enable_dhcp': True, 'network_id': u'a39da21a-e002-4a22-9416-33a2cabd4686', 'tenant_id': u'e356572fa5a14bed9f97250712903cac', 'dns_nameservers': [u'122.0.0.3'], 'gateway_ip': u'122.0.0.1', 'ip_version': 4L, 'shared': False}], 'enable_metadata_optimization': False, 'allowed_address_pairs': [], 'vrf_tenant': u'_noirolab_services', 'promiscuous_mode': True, 'ip_mapping': [{'nat_epg_name': u'NAT-epg-Datacenter-Out', 'external_segment_name': u'Datacenter-Out', 'nat_epg_tenant': 'common'}], 'floating_ip': [], 'host': u'fab102-compute-1.dmzlab.local', 'l3_policy_id': u'865b9d6c-6b6d-452d-baf5-f122f41902b4', 'vrf_subnets': [u'122.0.0.0/22', u'192.168.0.0/16'], 'device': u'4a48acd1-268c-4351-949b-06cf2ed390db', 'l2_policy_id': u'b2355b5d-4bf7-405b-a752-5492af4da971', 'enable_dhcp_optimization': True, 'port_id': u'4a48acd1-268c-4351-949b-06cf2ed390db', 'tenant_id': u'e356572fa5a14bed9f97250712903cac', 'app_profile_name': 'noirolab_app', 'vrf_name': 'advanced_services_monitoring_l3policy', 'mac_address': u'fa:16:3e:52:0f:fa', 'extra_ips': [], 'endpoint_group_name': 'Shd-L2POLICY.8ef17cae', 'ptg_tenant': u'_noirolab_services'}
2015-11-04 04:02:06.144 15352 ERROR oslo.messaging.rpc.dispatcher [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 ] Exception during message handling: Policy Target Group None could not be found
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/opflexagent/rpc.py", line 126, in get_gbp_details_list
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher for device in kwargs.pop('devices', [])
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/opflexagent/rpc.py", line 117, in get_gbp_details
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher return self.gbp_driver.get_gbp_details(context, **kwargs)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 223, in get_gbp_details
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher ptg, pt = self._port_id_to_ptg(context, port['id'])
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 1394, in _port_id_to_ptg
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher context, pt['policy_target_group_id']), pt
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/plugin.py", line 537, in get_policy_target_group
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher context, policy_target_group_id, None)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher return method(*args, **kwargs)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/gbpservice/neutron/db/grouppolicy/group_policy_db.py", line 1227, in get_policy_target_group
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher ptg = self._get_policy_target_group(context, policy_target_group_id)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/gbpservice/neutron/db/grouppolicy/group_policy_db.py", line 457, in _get_policy_target_group
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher policy_target_group_id=policy_target_group_id)
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher PolicyTargetGroupNotFound: Policy Target Group None could not be found
2015-11-04 04:02:06.144 15352 TRACE oslo.messaging.rpc.dispatcher
2015-11-04 04:02:06.144 15352 ERROR oslo.messaging._drivers.common [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 ] Returning exception Policy Target Group None could not be found to caller
2015-11-04 04:02:06.144 15352 ERROR oslo.messaging._drivers.common [req-b67eee01-595c-4e9c-a0e4-74c34f55e748 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/opflexagent/rpc.py", line 126, in get_gbp_details_list\n for device in kwargs.pop(\'devices\', [])\n', ' File "/usr/lib/python2.7/site-packages/opflexagent/rpc.py", line 117, in get_gbp_details\n return self.gbp_driver.get_gbp_details(context, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 223, in get_gbp_details\n ptg, pt = self._port_id_to_ptg(context, port[\'id\'])\n', ' File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 1394, in _port_id_to_ptg\n context, pt[\'policy_target_group_id\']), pt\n', ' File "/usr/lib/python2.7/site-packages/gbpservice/neutron/services/grouppolicy/plugin.py", line 537, in get_policy_target_group\n context, policy_target_group_id, None)\n', ' File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper\n return method(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/gbpservice/neutron/db/grouppolicy/group_policy_db.py", line 1227, in get_policy_target_group\n ptg = self._get_policy_target_group(context, policy_target_group_id)\n', ' File "/usr/lib/python2.7/site-packages/gbpservice/neutron/db/grouppolicy/group_policy_db.py", line 457, in _get_policy_target_group\n policy_target_group_id=policy_target_group_id)\n', 'PolicyTargetGroupNotFound: Policy Target Group None could not be found\n']
Error logs on opflex:
2015-11-04 04:01:37.311 3677 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] process_network_ports - iteration:2461 - failure while retrieving port details from server
2015-11-04 04:01:37.311 3677 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-11-04 04:01:37.311 3677 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1228, in process_network_ports
2015-11-04 04:01:37.311 3677 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent devices_added_updated, ovs_restarted)
2015-11-04 04:01:37.311 3677 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/opflexagent/gbp_ovs_agent.py", line 535, in treat_devices_added_or_updated
2015-11-04 04:01:37.311 3677 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise ovs.DeviceListRetrievalError(devices=devices, error=e)
2015-11-04 04:01:37.311 3677 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent DeviceListRetrievalError: Unable to retrieve port details for devices: set([u'0d78f42c-bf29-440c-a342-7a031d18edb1', u'ef51ce1f-cc91-4355-b3ed-630614f980d0', u'dcbb8970-1f3c-4c98-858d-cfffe42f4132', u'a3c33176-627e-432b-ad63-3e5d67840c72', u'dfe46462-4897-457b-b08a-2f0478c4728d', u'a4a2c02d-83e7-4779-97e4-ec5a6fb7ae57', u'aaa16bb1-13c1-4c0b-ac1d-4ac5180ec7d8', u'2928f354-4d43-4731-88ae-31084eba8068', u'24954338-7d23-4ba9-a9a9-40fc794d6d15', u'9f8d93d9-44b8-41f8-9dfc-9af2d04dfb5f', u'57735949-6f63-4bc1-994c-0033f53c7ae4', u'89eec669-2471-4b85-b4a4-2d9fd3f28658', u'aeaa6a13-6af3-4379-aab7-3c838e9d3788', u'6c5a3c4a-ee48-458e-8594-baa24df7332b', u'35dc334d-6368-499e-ab4c-4a1d4ddb1ec0', u'096fb9c3-75b5-47ea-8e34-e18b362f31c7', u'4a48acd1-268c-4351-949b-06cf2ed390db', u'e5cd64bc-418a-4a9c-8da6-4328d1971d60', u'5ac7c94c-ae81-42e6-8ba0-ec528f5ebb90', u'f052ef7e-f0a4-4cb8-a34e-61c67ee1e4b0', u'157b4694-ee09-451f-9280-dbdc97f374d4', u'0f107d08-2e2f-4a02-b002-62222f466084', u'7f0eec4b-c68b-4378-ab3f-f284953ebe9b', u'9ca6cbf2-8465-42e9-9fec-7f859fbc7882', u'5ac08c2e-9141-4e96-8221-26de7adb75de', u'6b7f0d08-bc09-44ea-930e-821ebff2306a', u'adf26c6d-02d4-487d-b351-b95623114031', u'76948cfe-0e35-478c-a334-c37e3bf21063', u'4c74142e-8d73-423c-888c-34cc2c7f99e5', u'c9943367-e887-4153-9060-611e8622939a', u'b90b986f-9b3d-48bb-b3b3-1f52bf5daf16', u'727c39c7-4915-49f9-a04a-f1df9bff970d', u'50118c15-9bec-4698-8125-b5c4bcf50d2e', u'47df799c-bb0a-45ee-9d26-636ad9304c38', u'a297e9b3-2a30-4ed0-88f5-3560d2fd0064', u'378948f7-b9c9-4480-a3d3-2fe416a38bcf', u'69a78469-0279-453d-a583-def867f437be', u'0bc507cd-4df7-4fc7-a346-098279a08bd8', u'd081c064-a04d-409d-9686-0d0c64e375aa', u'8f98f2d8-01eb-4642-a6e8-df4694891635', u'f7c83c8d-a270-4c01-b4ba-88fc7f66b5c0', u'3fdd2236-61d7-468a-960c-7906fd444d44', u'd4cbe8b5-d802-416e-97c6-de28a686d44a', u'ae362e8b-ea73-474f-93d2-5219c0cf0994', u'6b08c706-bd0f-4318-8b93-edd1b7d192ae', u'b0cf9ac0-6634-41fc-a126-7bad16aa4caa', u'4246cfaf-06f2-4768-b02c-c40dcc12ae98']) because of error: Remote error: PolicyTargetGroupNotFound Policy Target Group None could not be found
We should probably first investigate https:/