create network resource crashes with dynamic creds

[Ritesh Anand]

During tempest run, I noticed the following crash.
This is likely happening due to issuing a DELETE for router for which updates are still in progress.
Looks like a timing issue and may not happen all the time, but code crash is unexpected.

"setUpClass (designate_tempest_plugin.tests.api.v2.test_zones.ZonesTest)",
"-----------------------------------------------------------------------",
"",
"Captured traceback:",
"~~~~~~~~~~~~~~~~~~~",
" Traceback (most recent call last):",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/test.py\", line 172, in setUpClass",
" six.reraise(etype, value, trace)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/test.py\", line 157, in setUpClass",
" cls.setup_credentials()",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/test.py\", line 371, in setup_credentials",
" credential_type=credentials_type)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/test.py\", line 674, in get_client_manager",
" creds = getattr(cred_provider, credentials_method)()",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py\", line 353, in get_primary_creds",
" return self.get_credentials('primary')",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py\", line 345, in get_credentials",
" credentials.tenant_id)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py\", line 267, in _create_network_resources",
" self._clear_isolated_router(router['id'], router['name'])",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py\", line 378, in _clear_isolated_router",
" client.delete_router(router_id)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/services/network/routers_client.py\", line 52, in delete_router",
" return self.delete_resource(uri)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/services/network/base.py\", line 41, in delete_resource",
" resp, body = self.delete(req_uri)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/rest_client.py\", line 310, in delete",
" return self.request('DELETE', url, extra_headers, headers, body)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/rest_client.py\", line 668, in request",
" self._error_checker(resp, resp_body)",
" File \"/opt/stack/service/tempest/venv/lib/python2.7/site-packages/tempest/lib/common/rest_client.py\", line 789, in _error_checker",
" raise exceptions.Conflict(resp_body, resp=resp)",
" tempest.lib.exceptions.Conflict: Conflict with state of target resource",
" Details:
{u'message': u'Router 803c1ef7-ef63-48b5-bae0-577dd3e46c9e still has ports', u'type': u'RouterInUse', u'detail': u''}

[zhufl (zhu-fanglei)]

Thanks for reporting this.
I'm not sure what causes this, so let's look at it together.

The Tempest code:
        try:
            if not self.network_resources or self.network_resources['subnet']:
                subnet_name = rand_name_root + "-subnet"
                subnet = self._create_subnet(subnet_name, tenant_id,
                                             network['id'])
            if not self.network_resources or self.network_resources['router']:
                router_name = rand_name_root + "-router"
                router = self._create_router(router_name, tenant_id)
                self._add_router_interface(router['id'], subnet['id'])
        except Exception:
            try:
                if router:
                    self._clear_isolated_router(router['id'], router['name'])

From your traceback info we can see that something goes wrong after _create_router(maybe in _add_router_interface?) So, if _add_router_interface fails, then why _clear_isolated_router will get 'Router 803c1ef7-ef63-48b5-bae0-577dd3e46c9e still has ports' error?

So would you please dig more to find what causes router deletion error in your system?

[Slawek Kaplonski (slaweq)]

Do You have link to failed job? I would like to check logs from it

[Manik Bindlish (manikbindlish19)]

Any update on this?

[Martin Kopec (mkopec)]

Marking as Incomplete while waiting for the requested info.

[Launchpad Janitor (janitor)]

[Expired for tempest because there has been no activity for 60 days.]