[queens][fs017] telemetry tempest plugin tests failed in multinode promotion pipeline

[chandan kumar]

FS017 queens multinode periodic job started failing at while running telemetry tempest plugin
From the logs:
https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra//tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/b273262/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-18_06_36_27

2019-02-18 06:36:27 | {0} telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration.test_autoscaling [437.399938s] ... FAILED
2019-02-18 06:36:27 |
2019-02-18 06:36:27 | Captured traceback:
2019-02-18 06:36:27 | ~~~~~~~~~~~~~~~~~~~
2019-02-18 06:36:27 | Traceback (most recent call last):
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/telemetry_tempest_plugin/scenario/test_telemetry_integration.py", line 132, in test
2019-02-18 06:36:27 | self._do_test(filename)
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/telemetry_tempest_plugin/scenario/test_telemetry_integration.py", line 125, in _do_test
2019-02-18 06:36:27 | self.fail(msg)
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
2019-02-18 06:36:27 | raise self.failureException(msg)
2019-02-18 06:36:27 | AssertionError: From test "control stack status" :
2019-02-18 06:36:27 | _StringException: Traceback (most recent call last):
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/gabbi/suitemaker.py", line 95, in do_test
2019-02-18 06:36:27 | return test_method(*args, **kwargs)
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 94, in wrapper
2019-02-18 06:36:27 | func(self)
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 148, in test_request
2019-02-18 06:36:27 | self._run_test()
2019-02-18 06:36:27 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 551, in _run_test
2019-02-18 06:36:27 | raise failure
2019-02-18 06:36:27 | testtools.matchers._impl.MismatchError: 'UPDATE_COMPLETE' != u'UPDATE_FAILED': Unable to match $.stack.stack_status as UPDATE_COMPLETE, got UPDATE_FAILED

and this

2019-02-18 06:36:39 | tearDownClass (telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration)
2019-02-18 06:36:39 | -----------------------------------------------------------------------------------------------------
2019-02-18 06:36:39 |
2019-02-18 06:36:39 | Captured traceback:
2019-02-18 06:36:39 | ~~~~~~~~~~~~~~~~~~~
2019-02-18 06:36:39 | Traceback (most recent call last):
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 224, in tearDownClass
2019-02-18 06:36:39 | six.reraise(etype, value, trace)
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 196, in tearDownClass
2019-02-18 06:36:39 | teardown()
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 688, in clear_credentials
2019-02-18 06:36:39 | cls._creds_provider.clear_creds()
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 445, in clear_creds
2019-02-18 06:36:39 | self._clear_isolated_net_resources()
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 436, in _clear_isolated_net_resources
2019-02-18 06:36:39 | creds.subnet['name'])
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 386, in _clear_isolated_subnet
2019-02-18 06:36:39 | client.delete_subnet(subnet_id)
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
2019-02-18 06:36:39 | return self.delete_resource(uri)
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 41, in delete_resource
2019-02-18 06:36:39 | resp, body = self.delete(req_uri)
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
2019-02-18 06:36:39 | return self.request('DELETE', url, extra_headers, headers, body)
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
2019-02-18 06:36:39 | self._error_checker(resp, resp_body)
2019-02-18 06:36:39 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 789, in _error_checker
2019-02-18 06:36:39 | raise exceptions.Conflict(resp_body, resp=resp)
2019-02-18 06:36:39 | tempest.lib.exceptions.Conflict: Conflict with state of target resource
2019-02-18 06:36:39 | Details: {u'message': u'Unable to complete operation on subnet bbae2a4a-22de-4ca6-bed6-b4f7d45e9eda: One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}

^^ one is seen in master and rocky also:
https://bugs.launchpad.net/tripleo/+bug/1792862

need to investigate about both what brought this change.

[chandan kumar (chkumar246)]

https://github.com/gnocchixyz/gnocchi/pull/1019 -> getting backported to stable/4.2 gnocchi branch

[chandan kumar (chkumar246)]

For stack delete issue: https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra//tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/b273262/logs/subnode-2/var/log/containers/heat/heat-engine.log.txt.gz?level=ERROR

2019-02-18 06:30:47.716 10 ERROR heat.engine.resource Traceback (most recent call last):
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 921, in _action_recorder
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource yield
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1691, in update
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource prop_diff])
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 329, in wrapper
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource step = next(subtask)
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 975, in action_handler_task
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource handler_data = handler(*args)
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/aodh/gnocchi/alarm.py", line 125, in handle_update
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource self.client().alarm.update(self.resource_id, props)
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/aodhclient/v2/alarm.py", line 142, in update
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource data=jsonutils.dumps(alarm)).json()
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/aodhclient/v2/base.py", line 45, in _put
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource return self.client.api.put(*args, **kwargs)
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 313, in put
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource return self.request(url, 'PUT', **kwargs)
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/aodhclient/client.py", line 38, in request
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource raise exceptions.from_response(resp, url, method)
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource <html><head>
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource <title>500 Internal Server Error</title>
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource </head><body>
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource <h1>Internal Server Error</h1>
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource <p>The server encountered an internal error or
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource misconfiguration and was unable to complete
2019-02-18 06:30:47.716 10 ERROR heat.engine.resource your re...

[chandan kumar (chkumar246)]

https://github.com/openstack/telemetry-tempest-plugin/commit/7f0e315a78df17d981ed86ebddf87759cd97eedf will fix the issue, we need to create a new release for telemetry-tempest-plugin
and update the same in rdo.

[chandan kumar (chkumar246)]

Release telemetry-tempest-plugin 0.2.0 https://review.openstack.org/638118 and update the hash for queens https://review.rdoproject.org/r/#/c/18917/

[Quique Llorente (quiquell)]

Pending of https://github.com/gnocchixyz/gnocchi/pull/1019

[wes hayutin (weshayutin)]

http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/eb9395d/