While investigating https://bugs.launchpad.net/group-based-policy/+bug/1509458, I discovered that GroupPolicyDbPlugin._set_ess_for_l3p() gets called six times during creation of an L3Policy with an implicit ExternalSegment, and that two different IP addresses get passed to it for the same L3Policy. Only one IP address on the ExternalSegment should be needed (with NAT enabled). Its not clear if the extra address is being leaked or returned to the pool.
The following log with extra temporary log statements added shows the six _set_ess_for_l3p() calls, there callers, and the args passed:
2015-12-02 13:59:48,887 INFO [gbpservice.neutron.db.grouppolicy.group_policy_mapping_db] create_l3_policy() calling _set_ess_for_l3p()
2015-12-02 13:59:48,888 INFO [gbpservice.neutron.db.grouppolicy.group_policy_db] _set_ess_for_l3p - l3p_db: <gbpservice.neutron.db.grouppolicy.group_policy_mapping_db.L3PolicyMapping[object at 7c47690] {tenant_id='test-tenant', id='0fc541a0-6c04-4013-b8f4-8a3e7071dd07', type='mapping', name=u'l3p1', description=u'', ip_version=4, ip_pool=u'10.0.0.0/8', subnet_prefix_length=24, shared=False}>, es_dict: {}
2015-12-02 13:59:48,902 INFO [gbpservice.neutron.db.grouppolicy.group_policy_mapping_db] update_l3_policy() calling _set_ess_for_l3p()
2015-12-02 13:59:48,903 INFO [gbpservice.neutron.db.grouppolicy.group_policy_db] _set_ess_for_l3p - l3p_db: <gbpservice.neutron.db.grouppolicy.group_policy_mapping_db.L3PolicyMapping[object at 7b60750] {tenant_id=u'test-tenant', id=u'0fc541a0-6c04-4013-b8f4-8a3e7071dd07', type=u'mapping', name=u'l3p1', description=u'', ip_version=4, ip_pool=u'10.0.0.0/8', subnet_prefix_length=24, shared=False}>, es_dict: {u'e195b652-ed62-42f5-9810-8340bda908a5': []}
2015-12-02 13:59:48,967 WARNING [neutron.notifiers.nova] Authenticating to nova using nova_admin_* options is deprecated. This should be done using an auth plugin, like password
2015-12-02 13:59:49,105 INFO [neutron.callbacks.manager] Notify callbacks for port, after_create
2015-12-02 13:59:49,105 INFO [neutron.callbacks.manager] Calling callback neutron.db.l3_dvrscheduler_db._notify_l3_agent_new_port
2015-12-02 13:59:49,169 WARNING [neutron.scheduler.l3_agent_scheduler] No active L3 agents
2015-12-02 13:59:49,188 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] update_l3_policy_postcommit() calling _plug_l3p_to_es()
2015-12-02 13:59:49,188 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] _plug_l3p_to_es() calling set_external_fixed_ips()
2015-12-02 13:59:49,188 INFO [gbpservice.neutron.services.grouppolicy.group_policy_context] set_external_fixed_ips() calling _update_ess_for_l3p()
2015-12-02 13:59:49,190 INFO [gbpservice.neutron.db.grouppolicy.group_policy_mapping_db] _update_ess_for_l3p() calling _set_ess_for_l3p()
2015-12-02 13:59:49,190 INFO [gbpservice.neutron.db.grouppolicy.group_policy_db] _set_ess_for_l3p - l3p_db: <gbpservice.neutron.db.grouppolicy.group_policy_mapping_db.L3PolicyMapping[object at 7fbf9d0] {tenant_id=u'test-tenant', id=u'0fc541a0-6c04-4013-b8f4-8a3e7071dd07', type=u'mapping', name=u'l3p1', description=u'', ip_version=4, ip_pool=u'10.0.0.0/8', subnet_prefix_length=24, shared=False}>, es_dict: {u'e195b652-ed62-42f5-9810-8340bda908a5': [u'169.254.0.2']}
2015-12-02 13:59:49,197 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] _plug_l3p_to_es() calling _plug_l3p_to_es()
2015-12-02 13:59:49,197 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] _plug_l3p_to_es() calling set_external_fixed_ips()
2015-12-02 13:59:49,197 INFO [gbpservice.neutron.services.grouppolicy.group_policy_context] set_external_fixed_ips() calling _update_ess_for_l3p()
2015-12-02 13:59:49,199 INFO [gbpservice.neutron.db.grouppolicy.group_policy_mapping_db] _update_ess_for_l3p() calling _set_ess_for_l3p()
2015-12-02 13:59:49,199 INFO [gbpservice.neutron.db.grouppolicy.group_policy_db] _set_ess_for_l3p - l3p_db: <gbpservice.neutron.db.grouppolicy.group_policy_mapping_db.L3PolicyMapping[object at 7fbf9d0] {tenant_id=u'test-tenant', id=u'0fc541a0-6c04-4013-b8f4-8a3e7071dd07', type=u'mapping', name=u'l3p1', description=u'', ip_version=4, ip_pool=u'10.0.0.0/8', subnet_prefix_length=24, shared=False}>, es_dict: {u'e195b652-ed62-42f5-9810-8340bda908a5': [u'169.254.0.2']}
2015-12-02 13:59:49,242 WARNING [neutron.notifiers.nova] Authenticating to nova using nova_admin_* options is deprecated. This should be done using an auth plugin, like password
2015-12-02 13:59:49,375 INFO [neutron.callbacks.manager] Notify callbacks for port, after_create
2015-12-02 13:59:49,375 INFO [neutron.callbacks.manager] Calling callback neutron.db.l3_dvrscheduler_db._notify_l3_agent_new_port
2015-12-02 13:59:49,439 WARNING [neutron.scheduler.l3_agent_scheduler] No active L3 agents
2015-12-02 13:59:49,458 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] create_l3_policy_postcommit() calling _plug_l3p_to_es()
2015-12-02 13:59:49,458 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] _plug_l3p_to_es() calling set_external_fixed_ips()
2015-12-02 13:59:49,458 INFO [gbpservice.neutron.services.grouppolicy.group_policy_context] set_external_fixed_ips() calling _update_ess_for_l3p()
2015-12-02 13:59:49,460 INFO [gbpservice.neutron.db.grouppolicy.group_policy_mapping_db] _update_ess_for_l3p() calling _set_ess_for_l3p()
2015-12-02 13:59:49,460 INFO [gbpservice.neutron.db.grouppolicy.group_policy_db] _set_ess_for_l3p - l3p_db: <gbpservice.neutron.db.grouppolicy.group_policy_mapping_db.L3PolicyMapping[object at 873f210] {tenant_id=u'test-tenant', id=u'0fc541a0-6c04-4013-b8f4-8a3e7071dd07', type=u'mapping', name=u'l3p1', description=u'', ip_version=4, ip_pool=u'10.0.0.0/8', subnet_prefix_length=24, shared=False}>, es_dict: {u'e195b652-ed62-42f5-9810-8340bda908a5': [u'169.254.0.3']}
2015-12-02 13:59:49,466 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] _plug_l3p_to_es() calling _plug_l3p_to_es()
2015-12-02 13:59:49,466 INFO [gbpservice.neutron.services.grouppolicy.drivers.cisco.apic.apic_mapping] _plug_l3p_to_es() calling set_external_fixed_ips()
2015-12-02 13:59:49,467 INFO [gbpservice.neutron.services.grouppolicy.group_policy_context] set_external_fixed_ips() calling _update_ess_for_l3p()
2015-12-02 13:59:49,468 INFO [gbpservice.neutron.db.grouppolicy.group_policy_mapping_db] _update_ess_for_l3p() calling _set_ess_for_l3p()
2015-12-02 13:59:49,468 INFO [gbpservice.neutron.db.grouppolicy.group_policy_db] _set_ess_for_l3p - l3p_db: <gbpservice.neutron.db.grouppolicy.group_policy_mapping_db.L3PolicyMapping[object at 873f210] {tenant_id=u'test-tenant', id=u'0fc541a0-6c04-4013-b8f4-8a3e7071dd07', type=u'mapping', name=u'l3p1', description=u'', ip_version=4, ip_pool=u'10.0.0.0/8', subnet_prefix_length=24, shared=False}>, es_dict: {u'e195b652-ed62-42f5-9810-8340bda908a5': [u'169.254.0.3']}
I suspect the redundant _set_ess_for_l3p() calls, and the two different IP addresses, may be occurring because the implicit_policy driver is updating the L3P with the ES during its create_l3_policy_postcommit() by doing an update_l3_policy(), and this results in the apic_mapping driver's update_l3_policy_postcommit() running for the L3P before it's create_l3_policy_postcommit() runs. The apic_mapping driver's create_l3_policy_postcommit() needs to avoid processing the ES if it has already been processed by the driver's update_l3_policy_postcommit().
This bug addresses the two _set_ess_for_l3p() calls to which different IP addresses are initially passed. I'll file a separate bug regarding the redundant 2nd call to _set_ess_for_l3p() with each IP address.
> the implicit_policy driver is updating the L3P with the ES during its create_
This behavior of the implicit policy driver seems incorrect to me, for example an "update" call is made to the backend driver before a "create" call for a resource. Besides, it affects all backend drivers, including the resource-mapping driver. So instead of working around it in every backend driver, I'm proposing to fix it in implicit-policy driver by doing following in L3PolicyContext
* Don't call GroupPolicyPlug
* As before, update the L3PolicyContext with 'default' external-segment ID
Doing this will ensure that before the create post-commit of the backend drivers is called, both the context and DB will be set up in a way that would be the same as when the create L3Policy request referred to 'default' external segment.