[apic mapping] Create group failed due to overlapping IP on external segment

[Magesh GV]

I am using implicit workflow with the default external segment name set as DataCenter-Out in Neutron conf.

This issue occured once after using the test bed with groups being created and deleted successfully for a large number of times. The next group creation after this failure worked fine.

2015-10-23 10:23:55.693 1169 ERROR gbpservice.neutron.services.grouppolicy.policy_driver_manager [-] Policy driver 'apic' failed in create_l3_po
licy_postcommit
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager Traceback (most recent call last):
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpser
vice/neutron/services/grouppolicy/policy_driver_manager.py", line 119, in _call_on_drivers
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager getattr(driver.obj, method_name)(context)
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpser
vice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 603, in create_l3_policy_postcommit
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager self._plug_l3p_to_es(context, es)
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpser
vice/neutron/services/grouppolicy/drivers/cisco/apic/apic_mapping.py", line 1387, in _plug_l3p_to_es
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager context.set_external_fixed_ips(es['id'], [i
p])
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpser
vice/neutron/services/grouppolicy/group_policy_context.py", line 162, in set_external_fixed_ips
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager self._l3_policy['external_segments'])
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpser
vice/neutron/db/grouppolicy/group_policy_mapping_db.py", line 206, in _update_ess_for_l3p
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager self._set_ess_for_l3p(context, l3p_db, ess)
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager File "/usr/lib/python2.7/site-packages/gbpservice/neutron/db/grouppolicy/group_policy_db.py", line 820, in _set_ess_for_l3p
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager es_id=es['id'])
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager IpAddressOverlappingInExternalSegment: One or more requested IP addresses are already allocated for External Segment 4fcaaa43-386c-4aa1-bc4d-1a66da0964cd.
2015-10-23 10:23:55.693 1169 TRACE gbpservice.neutron.services.grouppolicy.policy_driver_manager
2015-10-23 10:23:55.699 1169 ERROR gbpservice.neutron.services.grouppolicy.plugin [-] create_l3_policy_postcommit failed, deleting l3_policy 5acf512d-5c47-4560-a8f7-d9deb57d98da

[Robert Kukura (rkukura)]

Was there concurrent activity at the point in which this occurred, or was everything serialized? If there was concurrent activity, can you provide details on what kinds of operations might have been processed concurrently? For example, were PTGs being created for multiple different tenants concurrently, or were PTG creates and deletes for the same tenant occurring concurrently?

[Magesh GV (magesh-gv)]

When I originally reported this issue, there were no concurrent creations on either the same tenant or on different tenants. I only created a PTG from the UI and it failed with this error.

I did not run into this issue after that, but going through neutron logs on one of the heavily used test beds today, I see this same error log about 10 times.

[Robert Kukura (rkukura)]

Without the ability to reproduce this issue, I've done all I can for now, and don't have any theory on why this failure is occurring. I did discover a couple of issues that are related to external segment IP allocation, but don't seem to be responsible for this error, and filed https://bugs.launchpad.net/group-based-policy/+bug/1522554 and https://bugs.launchpad.net/group-based-policy/+bug/1522613.