neutron.plugins.ml2.plugin error: "PortNotFound: Port xxxxx could not be found"

[Dennis Dmitriev]

OSTF test 'Check network connectivity from instance via floating IP' failed on CI test http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.centos.bonding_ha/66/ (deploy_bonding_ha_balance_slb):

====== ostf.log
AssertionError: Step 16 failed: Network can not be deleted Please refer to OpenStack logs for more details.

Cluster configuration:
- CentOS/HA
- Neutron/GRE
- networks on bonding interface (balance-slb) using eth1-eth4: public, management, storage.

- 3 controllers, 2 compute.

After run OSTF, primary controller (node-4) contains the following error of neutron.plugins.ml2.plugin:

===== node-4 neutron-server.log
2014-12-22T04:27:55.589194+00:00 err: 2014-12-22 04:27:53.183 4960 ERROR neutron.plugins.ml2.plugin [req-71a6306e-8050-4dcb-8e73-60923ed37954 None] Exception auto-deleting port ebd6e4ba-8332-41a7-baf9-4de8fe4023f1
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin Traceback (most recent call last):
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 651, in delete_network
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin self.delete_port(context, port.id)
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 985, in delete_port
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin l3plugin.prevent_l3_port_deletion(context, id)
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.6/site-packages/neutron/db/l3_db.py", line 881, in prevent_l3_port_deletion
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin port_db = self._core_plugin._get_port(context, port_id)
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 108, in _get_port
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin raise n_exc.PortNotFound(port_id=id)
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin PortNotFound: Port ebd6e4ba-8332-41a7-baf9-4de8fe4023f1 could not be found
2014-12-22 04:27:53.183 4960 TRACE neutron.plugins.ml2.plugin

# fuel --fuel-version
api: '1.0'
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-18_01-32-01
build_number: '56'
feature_groups:
- mirantis
fuellib_sha: 73332192a257ea02c40a39885c502ad1ebdf3eda
fuelmain_sha: 45caacadb878abfbd9d60e134d72229698b469c9
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
      build_id: 2014-12-18_01-32-01
      build_number: '56'
      feature_groups:
      - mirantis
      fuellib_sha: 73332192a257ea02c40a39885c502ad1ebdf3eda
      fuelmain_sha: 45caacadb878abfbd9d60e134d72229698b469c9
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
      production: docker
      release: '6.0'

[Eugene Nikanorov (enikanorov)]

There's some kind of race condition between deleting dhcp port initiated by subnet_delete and delete_network happening simultaneously at the end of ostf test.

As an immediate fix I can recommend removing delete_subnet from ostf test cleanup because delete_network does all the necessary cleanup.

[Eugene Nikanorov (enikanorov)]

Adding MOS project.

The issue should not show up too often, hence setting to medium importance.