Integrity violation on delete network

[Salvatore Orlando]

Found while running tests for bug 1224001.
Full logs here: http://logs.openstack.org/24/49424/13/check/check-tempest-devstack-vm-neutron-pg-isolated/405d3b4

Keeping to medium priority for now.
Will raise priority if we found more occurrences.

2013-10-04 21:20:46.888 31438 ERROR neutron.api.v2.resource [-] delete failed
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource Traceback (most recent call last):
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 84, in resource
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource result = method(request=request, **args)
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 432, in delete
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 411, in delete_network
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource break
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource self.commit()
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource self._prepare_impl()
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource self.session.flush()
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/openstack/common/db/sqlalchemy/session.py", line 542, in _wrap
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource raise exception.DBError(e)
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource DBError: (IntegrityError) update or delete on table "networks" violates foreign key constraint "ports_network_id_fkey" on table "ports"
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource DETAIL: Key (id)=(c63057f4-8d8e-497c-95d6-0d93d2cc83f5) is still referenced from table "ports".
2013-10-04 21:20:46.888 31438 TRACE neutron.api.v2.resource 'DELETE FROM networks WHERE networks.id = %(id)s' {'id': u'c63057f4-8d8e-497c-95d6-0d93d2cc83f5'}

[Joe Gordon (jogo)]

query: @message:"update or delete on table \"networks\" violates foreign key constraint \"ports_network_id_fkey\" on table \"ports\"" AND @fields.filename:"logs/screen-q-svc.txt"

found 10 hits in last 24 hours using logstash.openstack.org

[Henry Gessau (gessau)]

This might be fixed by https://review.openstack.org/49644 (line 395).

[Sean Dague (sdague)]

This is confirmed to still be an issue, just saw it fail again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/49644
Committed: http://github.com/openstack/neutron/commit/ed78b563e13f1ed9189d7c4b9cd4317f2a50e734
Submitter: Jenkins
Branch: master

commit ed78b563e13f1ed9189d7c4b9cd4317f2a50e734
Author: Bob Kukura <email address hidden>
Date: Thu Oct 3 12:25:24 2013 -0400

    Fix auto-deletion of ports when deleting subnets in ML2

    When a subnet is deleted, certain ports referencing it are
    auto-deleted. The implementation of NeutronDBPluginV2.delete_subnet()
    does this at the DB level, so ML2's mechanism drivers were not being
    called.

    Ml2Plugin.delete_subnet() is changed to not use the base class's
    method, and to auto-delete ports by calling its own delete_port()
    method outside of the transaction. A loop avoids race conditions with
    ports being asynchronously added to the subnet.

    The logic in Ml2Plugin.delete_network() is also fixed to properly
    handle auto-deleting ports and subnets, and debug logging is added to
    the various delete methods.

    Closes-Bug: 1234195
    Partial-Bug: 1235486
    Change-Id: I6d74f89d39ea8afe6915f1d2f9afdf66c0076f5a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/51303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51303
Committed: http://github.com/openstack/neutron/commit/bfe1dca0563cd5beadf91e3688924c44f5063778
Submitter: Jenkins
Branch: milestone-proposed

commit bfe1dca0563cd5beadf91e3688924c44f5063778
Author: Bob Kukura <email address hidden>
Date: Thu Oct 3 12:25:24 2013 -0400

    Fix auto-deletion of ports when deleting subnets in ML2

    When a subnet is deleted, certain ports referencing it are
    auto-deleted. The implementation of NeutronDBPluginV2.delete_subnet()
    does this at the DB level, so ML2's mechanism drivers were not being
    called.

    Ml2Plugin.delete_subnet() is changed to not use the base class's
    method, and to auto-delete ports by calling its own delete_port()
    method outside of the transaction. A loop avoids race conditions with
    ports being asynchronously added to the subnet.

    The logic in Ml2Plugin.delete_network() is also fixed to properly
    handle auto-deleting ports and subnets, and debug logging is added to
    the various delete methods.

    Closes-Bug: 1234195
    Partial-Bug: 1235486
    Change-Id: I6d74f89d39ea8afe6915f1d2f9afdf66c0076f5a
    (cherry picked from commit ed78b563e13f1ed9189d7c4b9cd4317f2a50e734)

[Anita Kuno (anteaya)]

Failures continue to be attributed to this bug: http://logstash.openstack.org/#eyJmaWVsZHMiOiBbXSwgInNlYXJjaCI6ICJtZXNzYWdlOlwidXBkYXRlIG9yIGRlbGV0ZSBvbiB0YWJsZSBcXFwibmV0d29ya3NcXFwiIHZpb2xhdGVzIGZvcmVpZ24ga2V5IGNvbnN0cmFpbnRcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1xLXN2Yy50eHRcIlxuIiwgInRpbWVmcmFtZSI6ICI2MDQ4MDAiLCAiZ3JhcGhtb2RlIjogImNvdW50IiwgIm9mZnNldCI6IDB9

[Anita Kuno (anteaya)]

As of right now, failures seem to have stopped on the 16th. Let's keep an eye on this one.

[Armando Migliaccio (armando-migliaccio)]

Hopefully these ones killed it:

https://review.openstack.org/#/c/54850/
https://review.openstack.org/#/c/56496/