rally create_and_list_floating_ips task fails with duplicate IpamAllocation

[Huang Cheng]

Description
===========

environment
-----------

* RDO OpenStack Ocata
* neutron conf:
 service_plugins = router
 ml2 mechanism_drivers = openvswitch
 external network ex-network with a subnet 111.0.0.0/16
* rally 0.9.1
create_and_list_floating_ips.json
{
    "NeutronNetworks.create_and_list_floating_ips": [
        {
            "args": {
                "floating_network": "ex-network",
                "floating_ip_args": {}
            },
            "runner": {
                "type": "constant",
                "times": 500,
                "concurrency": 100
            },
            "context": {
                "users": {
                    "tenants": 2,
                    "users_per_tenant": 3
                },
                "quotas": {
                    "neutron": {
                        "floatingip": -1
                    }
                }
            },
            "sla": {
                "failure_rate": {
                    "max": 0
                }
            }
        }
    ]
}

rally result
------------

Total durations
Action Min (sec) Median (sec) 90%ile (sec) 95%ile (sec) Max (sec) Avg (sec) Success Count
neutron.create_floating_ip 4.2 18.946 71.561 91.803 131.663 29.769 94.2% 500
-> neutron.list_networks 0.991 1.837 10.769 11.56 12.955 3.366 94.2% 500
neutron.list_floating_ips 0.162 0.95 1.672 1.884 3.009 1.007 100.0% 471
total 4.396 20.196 71.896 92.886 131.663 30.717 94.2% 500
-> duration 4.396 20.196 71.896 92.886 131.663 30.717 94.2% 500
-> idle_duration 0 0 0 0 0 0 94.2% 500

html report here: https://pastebin.com/9bLJVhfS

neutron-server log
------------------

2017-10-21 10:50:50.445 20020 ERROR oslo_db.api [req-f4c8db2c-d80d-475a-b408-6eafd6701b62 07a6aba4b1a44dbc9453ddddbfbe4a65 bb0f198d854c440d8f7c0beb27eaae2b - - -] DB exceeded retry limit.
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api Traceback (most recent call last):
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api return f(*args, **kwargs)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api traceback.format_exc())
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api self.force_reraise()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api return f(*dup_args, **dup_kwargs)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1192, in create_port
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api result, mech_context = self._create_port_db(context, port)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1163, in _create_port_db
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api port_db = self.create_port_db(context, port)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1221, in create_port_db
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api context, port, port_id)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 191, in allocate_ips_for_port_and_store
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api revert_on_fail=False)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api self.force_reraise()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 172, in allocate_ips_for_port_and_store
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api ips = self._allocate_ips_for_port(context, port_copy)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 227, in _allocate_ips_for_port
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api return self._ipam_allocate_ips(context, ipam_driver, p, ips)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 147, in _ipam_allocate_ips
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api "external system for %s"), addresses)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api self.force_reraise()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/db/ipam_pluggable_backend.py", line 121, in _ipam_allocate_ips
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api ip_address, subnet_id = ipam_allocator.allocate(ip_request)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/ipam/subnet_alloc.py", line 232, in allocate
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api return ipam_subnet.allocate(address_request), subnet_id
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 210, in allocate
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api ip_address)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 127, in create_allocation
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api ipam_subnet_id=self._ipam_subnet_id).create()
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api res = func(self, *args, **kwargs)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 582, in create
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api object_class=self.__class__, db_exception=db_exc)
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api NeutronDbObjectDuplicateEntry: Failed to create a duplicate IpamAllocation: for attribute(s) ['PRIMARY'] with value(s) 111.0.0.204-f9f8245b-6920-4a51-a21b-76e0ab8e2984
2017-10-21 10:50:50.445 20020 ERROR oslo_db.api

[Boden R (boden)]

This maybe manifesting elsewhere as well [1]. If so it cropped up around 10/17/2017.

[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22NeutronDbObjectDuplicateEntry%3A%20Failed%20to%20create%20a%20duplicate%20IpamAllocation%3A%20for%20attribute(s)%20%5B'PRIMARY'%5D%5C%22

[Brian Haley (brian-haley)]

I don't see this error in any recent rally runs, so will close. Please re-open if you see it again.