Multiple network related tempest tests got failed in featureset 020 on stein
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/69f2259/logs/undercloud/home/zuul/tempest.log.txt.gz

neutron_tempest_plugin.api.test_revisions.TestRevisions.test_update_dns_domain_bumps_revision[id-40ba648f-f374-4c29-a5b7-489dd5a38a4e]_StringException: pythonlogging:'': {{{

-12-02 08:54:56,203 78 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '204', u'content-length': '0', 'content-location': 'http://10.0.0.5:9696/v2.0/networks/01386ec0-7604-499e-b671-424e66d722e8', u'date': 'Mon, 02 Dec 2019 08:54:56 GMT', u'connection': 'close', u'x-openstack-request-id': 'req-392ab90f-80c9-4173-9fde-9c76f33217bf'}
        Body:
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 108, in wrapper
    return func(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/api/test_revisions.py", line 200, in test_update_dns_domain_bumps_revision
    net['revision_number'])
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 1233, in assertGreater
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: 2 not greater than 2


ft2.1: neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest.test_dns_domain_and_name[id-988347de-07af-471a-abfa-65aea9f452a6]_StringException: pythonlogging:'': {{{

019-12-02 09:13:08,724 80 INFO     [tempest.lib.common.rest_client] Request (InternalDNSTest:_run_cleanups): 404 GET http://10.0.0.5:8774/v2.1/servers/c74784b2-764d-407f-ab3e-8c2f4ffc60bf 0.054s
2019-12-02 09:13:08,726 80 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '404', u'content-length': '111', 'content-location': 'http://10.0.0.5:8774/v2.1/servers/c74784b2-764d-407f-ab3e-8c2f4ffc60bf', u'x-compute-request-id': 'req-136bec23-81f4-4bfa-9ea8-1821d297aee0', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Mon, 02 Dec 2019 09:13:08 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'req-136bec23-81f4-4bfa-9ea8-1821d297aee0'}
        Body: {"itemNotFound": {"message": "Instance c74784b2-764d-407f-ab3e-8c2f4ffc60bf could not be found.", "code": 404}}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 108, in wrapper
    return func(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/test_internal_dns.py", line 75, in test_dns_domain_and_name
    self.assertIn('openstackgate.local', resolv_conf)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 356, in assertIn
    self.assertThat(haystack, Contains(needle), message)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'openstackgate.local' not in u'nameserver 1.1.1.1\nnameserver 1.0.0.1\nnameserver 38.145.33.91\n'

 setUpClass (neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests)_StringExceptio

 File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 571, in base_url
    endpoint_type, catalog))
tempest.lib.exceptions.EndpointNotFound: Endpoint not found
Details: No matching service found in the catalog.
Scope: project, Credentials: {'username': u'tempest-DNSIntegrationTests-598134849', 'project_name': u'tempest-DNSIntegrationTests-598134849', 'project_domain_id': u'default', 'user_domain_id': u'default', 'tenant_id': u'779c91d5b46c434db68c735e567f7024', 'user_domain_name': u'Default', 'domain_name': u'Default', 'tenant_name': u'tempest-DNSIntegrationTests-598134849', 'user_id': u'c753a799366641abb9478bdef5c1f7af', 'project_id': u'779c91d5b46c434db68c735e567f7024', 'domain_id': u'default', 'project_domain_name': u'Default'}
Auth data: {u'is_domain': False, u'methods': [u'password'], u'roles': [{u'id': u'2ae4baa65fd747d588f14f0b36508216', u'name': u'_member_'}], u'expires_at': u'2019-12-02T10:04:04.000000Z', u'project': {u'domain': {u'id': u'default', u'name': u'Default'}, u'id': u'779c91d5b46c434db68c735e567f7024', u'name': u'tempest-DNSIntegrationTests-598134849'}, u'catalog': [{u'endpoints': [{u'url': u'http://172.17.0.178:8774/v2.1', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'8d16132c1b914caa92f4bf8dbfa6f088'}, {u'url': u'http://10.0.0.5:8774/v2.1', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'c10e4e2364ea441e830cd6d12f84b6c9'}, {u'url': u'http://172.17.0.178:8774/v2.1', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'e4e3cc60876f4a20a643aef2e0d5fc6a'}], u'type': u'compute', u'id': u'1bcf17f8bc9f4e24b39e56a62d01e074', u'name': u'nova'}, {u'endpoints': [{u'url': u'http://172.17.0.178:9696', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'5cf9e5cea5084bd29f3875f9ab199347'}, {u'url': u'http://10.0.0.5:9696', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6b745aacd0754457a966ccbd5f2d4567'}, {u'url': u'http://172.17.0.178:9696', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'dd6e5cadda134c409776c90ee3fcc4a7'}], u'type': u'network', u'id': u'1dbbc8e7f0b8464c8620ca09f5dbf332', u'name': u'neutron'}, {u'endpoints': [{u'url': u'http://172.17.0.178:8778/placement', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'0c7209a8aeac4d769184a0436e6cf943'}, {u'url': u'http://172.17.0.178:8778/placement', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'90da2daced024c7a8e31e717d429a260'}, {u'url': u'http://10.0.0.5:8778/placement', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'f2477bdc1d154cbba637611c67234219'}], u'type': u'placement', u'id': u'250fdd6f9faa449fb78eb89122f610a2', u'name': u'placement'}, {u'endpoints': [{u'url': u'http://10.0.0.5:5000', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'10d318d38c2e45e08ae5ad94241daf78'}, {u'url': u'http://172.17.0.178:5000', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'278d1be2154d48349b8f6a593845ef83'}, {u'url': u'http://192.168.24.9:35357', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'58e77143029849118d46d1c541d8f2ec'}], u'type': u'identity', u'id': u'49040859306e4aa58d64d6ea43b250af', u'name': u'keystone'}, {u'endpoints': [{u'url': u'http://172.18.0.205:8080', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'b36794e0839e4d70a64f46e4af117817'}, {u'url': u'http://172.18.0.205:8080/v1/AUTH_779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'd3218bbc3ffe460986b936192ddff4e7'}, {u'url': u'http://10.0.0.5:8080/v1/AUTH_779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'dd1317abee544ac9ae345a0dadecc9cf'}], u'type': u'object-store', u'id': u'55bd1284389b468cb46034639c2fe021', u'name': u'swift'}, {u'endpoints': [{u'url': u'http://172.17.0.178:8776/v3/779c91d5b46c434db68c735e567f7024', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'00f1d4c05e9e4bf6b708907b73d7de78'}, {u'url': u'http://172.17.0.178:8776/v3/779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'044953e5169a4b43a6db4e20f83ad051'}, {u'url': u'http://10.0.0.5:8776/v3/779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'056b31f0ba11409c873c2751f8b0d404'}], u'type': u'volumev3', u'id': u'6e740f520afe40d18438943185d25920', u'name': u'cinderv3'}, {u'endpoints': [{u'url': u'http://10.0.0.5:8004/v1/779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'2d56d7618fc3470fb511446d7bd8aa80'}, {u'url': u'http://172.17.0.178:8004/v1/779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'bb9aa1c8c71d48fba8fac2823771b47f'}, {u'url': u'http://172.17.0.178:8004/v1/779c91d5b46c434db68c735e567f7024', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'd810bffa8d3a4d59b740724a5cef7538'}], u'type': u'orchestration', u'id': u'79f69c6feada41e8a146946e30c7256d', u'name': u'heat'}, {u'endpoints': [{u'url': u'http://172.17.0.178:9292', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'29fb4e757f364b53a033aeb9026b16be'}, {u'url': u'http://10.0.0.5:9292', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'44fc7402b36140d29d71e9d45322b975'}, {u'url': u'http://172.17.0.178:9292', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'b7955e2a058e42fd8f4583480fd53e29'}], u'type': u'image', u'id': u'b15563caca9848bfaee273ec6e22fa12', u'name': u'glance'}, {u'endpoints': [{u'url': u'http://10.0.0.5:8776/v2/779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6547e8581db048fd87bfb277afe505f3'}, {u'url': u'http://172.17.0.178:8776/v2/779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'682cd5cf4f7a4f109d5b63950917966d'}, {u'url': u'http://172.17.0.178:8776/v2/779c91d5b46c434db68c735e567f7024', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'98644f89aebc4b14865c14bcb31b950d'}], u'type': u'volumev2', u'id': u'd94ec47426ec4b92a31e61a65767221f', u'name': u'cinderv2'}, {u'endpoints': [{u'url': u'http://172.17.0.178:8000/v1', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'1f1debcf96de411f8a7ee2679b23248c'}, {u'url': u'http://10.0.0.5:8000/v1', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'456435210e434458b0f9763a5c04a52e'}, {u'url': u'http://172.17.0.178:8000/v1', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'524132a7d458460bbe467fdb507e11e0'}], u'type': u'cloudformation', u'id': u'eeba91b443ab4bdfb1145b10ed852fcd', u'name': u'heat-cfn'}], u'user': {u'domain': {u'id': u'default', u'name': u'Default'}, u'password_expires_at': None, u'name': u'tempest-DNSIntegrationTests-598134849', u'id': u'c753a799366641abb9478bdef5c1f7af'}, u'audit_ids': [u'IGCxitp3S167-MQ4OBJXOw'], u'issued_at': u'2019-12-02T09:04:04.000000Z'}
Service: dns, Region: regionOne, endpoint_type: public
Catalog: [{u'endpoints': [{u'url': u'http://172.17.0.178:8774/v2.1', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'8d16132c1b914caa92f4bf8dbfa6f088'}, {u'url': u'http://10.0.0.5:8774/v2.1', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'c10e4e2364ea441e830cd6d12f84b6c9'}, {u'url': u'http://172.17.0.178:8774/v2.1', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'e4e3cc60876f4a20a643aef2e0d5fc6a'}], u'type': u'compute', u'id': u'1bcf17f8bc9f4e24b39e56a62d01e074', u'name': u'nova'}, {u'endpoints': [{u'url': u'http://172.17.0.178:9696', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'5cf9e5cea5084bd29f3875f9ab199347'}, {u'url': u'http://10.0.0.5:9696', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6b745aacd0754457a966ccbd5f2d4567'}, {u'url': u'http://172.17.0.178:9696', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'dd6e5cadda134c409776c90ee3fcc4a7'}], u'type': u'network', u'id': u'1dbbc8e7f0b8464c8620ca09f5dbf332', u'name': u'neutron'}, {u'endpoints': [{u'url': u'http://172.17.0.178:8778/placement', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'0c7209a8aeac4d769184a0436e6cf943'}, {u'url': u'http://172.17.0.178:8778/placement', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'90da2daced024c7a8e31e717d429a260'}, {u'url': u'http://10.0.0.5:8778/placement', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'f2477bdc1d154cbba637611c67234219'}], u'type': u'placement', u'id': u'250fdd6f9faa449fb78eb89122f610a2', u'name': u'placement'}, {u'endpoints': [{u'url': u'http://10.0.0.5:5000', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'10d318d38c2e45e08ae5ad94241daf78'}, {u'url': u'http://172.17.0.178:5000', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'278d1be2154d48349b8f6a593845ef83'}, {u'url': u'http://192.168.24.9:35357', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'58e77143029849118d46d1c541d8f2ec'}], u'type': u'identity', u'id': u'49040859306e4aa58d64d6ea43b250af', u'name': u'keystone'}, {u'endpoints': [{u'url': u'http://172.18.0.205:8080', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'b36794e0839e4d70a64f46e4af117817'}, {u'url': u'http://172.18.0.205:8080/v1/AUTH_779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'd3218bbc3ffe460986b936192ddff4e7'}, {u'url': u'http://10.0.0.5:8080/v1/AUTH_779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'dd1317abee544ac9ae345a0dadecc9cf'}], u'type': u'object-store', u'id': u'55bd1284389b468cb46034639c2fe021', u'name': u'swift'}, {u'endpoints': [{u'url': u'http://172.17.0.178:8776/v3/779c91d5b46c434db68c735e567f7024', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'00f1d4c05e9e4bf6b708907b73d7de78'}, {u'url': u'http://172.17.0.178:8776/v3/779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'044953e5169a4b43a6db4e20f83ad051'}, {u'url': u'http://10.0.0.5:8776/v3/779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'056b31f0ba11409c873c2751f8b0d404'}], u'type': u'volumev3', u'id': u'6e740f520afe40d18438943185d25920', u'name': u'cinderv3'}, {u'endpoints': [{u'url': u'http://10.0.0.5:8004/v1/779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'2d56d7618fc3470fb511446d7bd8aa80'}, {u'url': u'http://172.17.0.178:8004/v1/779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'bb9aa1c8c71d48fba8fac2823771b47f'}, {u'url': u'http://172.17.0.178:8004/v1/779c91d5b46c434db68c735e567f7024', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'd810bffa8d3a4d59b740724a5cef7538'}], u'type': u'orchestration', u'id': u'79f69c6feada41e8a146946e30c7256d', u'name': u'heat'}, {u'endpoints': [{u'url': u'http://172.17.0.178:9292', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'29fb4e757f364b53a033aeb9026b16be'}, {u'url': u'http://10.0.0.5:9292', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'44fc7402b36140d29d71e9d45322b975'}, {u'url': u'http://172.17.0.178:9292', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'b7955e2a058e42fd8f4583480fd53e29'}], u'type': u'image', u'id': u'b15563caca9848bfaee273ec6e22fa12', u'name': u'glance'}, {u'endpoints': [{u'url': u'http://10.0.0.5:8776/v2/779c91d5b46c434db68c735e567f7024', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6547e8581db048fd87bfb277afe505f3'}, {u'url': u'http://172.17.0.178:8776/v2/779c91d5b46c434db68c735e567f7024', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'682cd5cf4f7a4f109d5b63950917966d'}, {u'url': u'http://172.17.0.178:8776/v2/779c91d5b46c434db68c735e567f7024', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'98644f89aebc4b14865c14bcb31b950d'}], u'type': u'volumev2', u'id': u'd94ec47426ec4b92a31e61a65767221f', u'name': u'cinderv2'}, {u'endpoints': [{u'url': u'http://172.17.0.178:8000/v1', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'1f1debcf96de411f8a7ee2679b23248c'}, {u'url': u'http://10.0.0.5:8000/v1', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'456435210e434458b0f9763a5c04a52e'}, {u'url': u'http://172.17.0.178:8000/v1', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'524132a7d458460bbe467fdb507e11e0'}], u'type': u'cloudformation', u'id': u'eeba91b443ab4bdfb1145b10ed852fcd', u'name': u'heat-cfn'}]

tempest.api.network.admin.test_routers.RoutersAdminTest.test_create_router_with_snat_explicit[id-ea74068d-09e9-4fd7-8995-9b6a1ace920f]_Str

    Response - Headers: {'status': '500', u'content-length': '150', 'content-location': 'http://10.0.0.5:9696/v2.0/routers/999a82de-4b0d-4e5d-851c-bf07d84f452d', u'date': 'Mon, 02 Dec 2019 08:03:41 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-65833112-d708-47f3-8227-3261cfda5af2'}
        Body: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/routers_client.py", line 52, in delete_router
    return self.delete_resource(uri)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 314, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 679, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 856, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.

Few failed during cleanup, It is a tracker bug, we will be putting more info for each failed tests.