DBDuplicateEntry: (IntegrityError) duplicate key value violates unique constraint "networkdhcpagentbindings_pkey"

[Attila Fazekas]

Exception:
http://logs.openstack.org/53/53653/2/check/check-tempest-devstack-vm-neutron-pg-isolated/06e0bbe/logs/screen-q-svc.txt.gz#_2013-10-25_18_16_06_688

2013-10-25 18:16:06.688 1299 ERROR neutron.api.v2.resource [-] create failed
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource Traceback (most recent call last):
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 84, in resource
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource result = method(request=request, **args)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 407, in create
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource obj)})
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 386, in notify
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource notifier_method)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 268, in _send_dhcp_notification
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self._dhcp_agent_notifier.notify(context, data, methodname)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 132, in notify
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self._notification(context, methodname, data, network_id)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 72, in _notification
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource chosen_agents = plugin.schedule_network(adminContext, network)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/agentschedulers_db.py", line 207, in schedule_network
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self, context, created_network)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/scheduler/dhcp_agent_scheduler.py", line 83, in schedule
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self._schedule_bind_network(context, agent, network['id'])
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self.commit()
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self._prepare_impl()
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource self.session.flush()
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/openstack/common/db/sqlalchemy/session.py", line 541, in _wrap
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource _raise_if_duplicate_entry_error(e, get_engine().name)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/openstack/common/db/sqlalchemy/session.py", line 492, in _raise_if_duplicate_entry_error
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource raise exception.DBDuplicateEntry(columns, integrity_error)
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource DBDuplicateEntry: (IntegrityError) duplicate key value violates unique constraint "networkdhcpagentbindings_pkey"
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource DETAIL: Key (network_id, dhcp_agent_id)=(f89ed027-b31b-4eeb-a5db-7e42a90c3641, 412ef532-fc08-4ffe-9fbd-a1bd0b1c0b04) already exists.
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource 'INSERT INTO networkdhcpagentbindings (network_id, dhcp_agent_id) VALUES (%(network_id)s, %(dhcp_agent_id)s)' {'network_id': u'f89ed027-b31b-4eeb-a5db-7e42a90c3641', 'dhcp_agent_id': u'412ef532-fc08-4ffe-9fbd-a1bd0b1c0b04'}
2013-10-25 18:16:06.688 1299 TRACE neutron.api.v2.resource

2013-10-25 18:24:58.500 | ======================================================================
2013-10-25 18:24:58.500 | FAIL: setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsTestXML)
2013-10-25 18:24:58.500 | setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsTestXML)
2013-10-25 18:24:58.501 | ----------------------------------------------------------------------
2013-10-25 18:24:58.501 | _StringException: Traceback (most recent call last):
2013-10-25 18:24:58.501 | File "tempest/api/compute/servers/test_server_actions.py", line 53, in setUpClass
2013-10-25 18:24:58.502 | cls.rebuild_server()
2013-10-25 18:24:58.502 | File "tempest/api/compute/base.py", line 95, in rebuild_server
2013-10-25 18:24:58.502 | resp, server = cls.create_server(wait_until='ACTIVE', **kwargs)
2013-10-25 18:24:58.502 | File "tempest/api/compute/base.py", line 143, in create_server
2013-10-25 18:24:58.503 | server['id'], kwargs['wait_until'])
2013-10-25 18:24:58.503 | File "tempest/services/compute/xml/servers_client.py", line 356, in wait_for_server_status
2013-10-25 18:24:58.503 | return waiters.wait_for_server_status(self, server_id, status)
2013-10-25 18:24:58.504 | File "tempest/common/waiters.py", line 71, in wait_for_server_status
2013-10-25 18:24:58.504 | raise exceptions.BuildErrorException(server_id=server_id)
2013-10-25 18:24:58.504 | BuildErrorException: Server 6fe57814-4866-4789-ba68-b0ab5acd3956 failed to build and is in ERROR status

[Armando Migliaccio (armando-migliaccio)]

In another manifestation of this bug I saw this stack trace:

http://paste.openstack.org/show/65146/

this leads me to believe the session is not rolled back correctly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/73386

[Eugene Nikanorov (enikanorov)]

I wonder is both of this reproducible after https://review.openstack.org/#/c/71393/ has been merged?

[Armando Migliaccio (armando-migliaccio)]

If anything else change https://review.openstack.org/#/c/71393/ might have made things worse! See bug: 1282421

[Armando Migliaccio (armando-migliaccio)]

Sorry bug #1282421

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73386
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cb3f86dd781587eaa9fca6cc4a2f98de93526db7
Submitter: Jenkins
Branch: master

commit cb3f86dd781587eaa9fca6cc4a2f98de93526db7
Author: armando-migliaccio <email address hidden>
Date: Thu Feb 13 11:36:34 2014 -0800

    Ensure that session is rolled back on bulk creates

    During bulk creates, the session is began explicitely;
    ensure that it gets rolled back before re-raising in
    order to avoid triggering InvalidRequestError
    exceptions when the session is reused.

    Partial-bug: 1244757

    Change-Id: I797faeec2ca5374620db905599ab6e6f04fdcabd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/104246

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104246
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e4bdc927fa54e3e50ec0ed5bc5791362a501ff4
Submitter: Jenkins
Branch: stable/havana

commit 8e4bdc927fa54e3e50ec0ed5bc5791362a501ff4
Author: armando-migliaccio <email address hidden>
Date: Thu Feb 13 11:36:34 2014 -0800

    Ensure that session is rolled back on bulk creates

    During bulk creates, the session is began explicitely;
    ensure that it gets rolled back before re-raising in
    order to avoid triggering InvalidRequestError
    exceptions when the session is reused.

    Partial-bug: 1244757

    Change-Id: I797faeec2ca5374620db905599ab6e6f04fdcabd
    (cherry picked from commit cb3f86dd781587eaa9fca6cc4a2f98de93526db7)