telemetry_tempest_plugin.scenario.test_telemetry_integration is failing on featureset 017

[Arx Cruz]

Logs:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/679b746/undercloud/home/jenkins/tempest/tempest.html.gz

Failure:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/telemetry_tempest_plugin/scenario/test_telemetry_integration.py", line 132, in test
    self._do_test(filename)
  File "/usr/lib/python2.7/site-packages/telemetry_tempest_plugin/scenario/test_telemetry_integration.py", line 125, in _do_test
    self.fail(msg)
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: From test "control stack status" :
_StringException: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 93, in wrapper
    func(self)
  File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 141, in test_request
    self._run_test()
  File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 463, in _run_test
    raise failure
testtools.matchers._impl.MismatchError: 'CREATE_COMPLETE' != u'CREATE_FAILED': Unable to match $.stack.stack_status as CREATE_COMPLETE, got CREATE_FAILED

It seems to me that is resource issue, since this particular job also runs 40 other tests, I'll reduce the number of concurrency and see if it will work.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/547070

[Mehdi Abaakouk (sileht)]

Resume of my investigation:

The Heat ERROR shows a Gnocchi API 500

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/679b746/subnode-2/var/log/containers/heat/heat-engine.log.txt.gz#_2018-02-22_12_35_23_846

The Gnocchi API ERROR shows that swiftclient returns None instead of a tuple

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/679b746/subnode-2/var/log/extra/docker/containers/gnocchi_api/log/httpd/gnocchi_wsgi_error.log.txt.

Swiftclient doc is explicit about the returns, it should be a tuple or ClientException exception

https://docs.openstack.org/python-swiftclient/latest/swiftclient.html#swiftclient.client.get_container

Swift log are missing to continue the investigation

So we have at least two bugs here:
* a swiftclient bug that return None in some circonstency instead of the expected tuple or expection
* a swift issue that triggers the client bug but we can't investigate it because swift proxy logs are missing

[Mehdi Abaakouk (sileht)]

I have opened for swiftclient https://bugs.launchpad.net/python-swiftclient/+bug/1752277

[Christian Schwede (cschwede)]

Looking at https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/679b746/undercloud/home/jenkins/tempest/tempest.html.gz I don't see Swift failures, but networking failures?

I wasn't able to reproduce a None return (yet) with python-swiftclient. As written in the LP bug from Mehdi, the Swift logs are missing so I can't investigate this further atm.

[wes hayutin (weshayutin)]

https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/

passed 6 times in a row on queens.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Arx Cruz (<email address hidden>) on branch: master
Review: https://review.openstack.org/547070