CI: telemetry tempest tests fail for scenario 001 in periodic jobs

[Sagi (Sergey) Shnaidman]

periodic-tripleo-ci-centos-7-scenario001-multinode-oooq fails:

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario001-multinode-oooq/cb596ff/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-09-03_08_02_22

2017-09-03 08:02:12 | {0} ceilometer.tests.tempest.scenario.test_telemetry_integration.TestTelemetryIntegration.test_autoscaling [915.185832s] ... FAILED
2017-09-03 08:02:22 | {0} tearDownClass (ceilometer.tests.tempest.scenario.test_telemetry_integration.TestTelemetryIntegration) [0.000000s] ... FAILED

2017-09-03 08:02:22 | ceilometer.tests.tempest.scenario.test_telemetry_integration.TestTelemetryIntegration.test_autoscaling
2017-09-03 08:02:22 | ------------------------------------------------------------------------------------------------------
2017-09-03 08:02:22 |
2017-09-03 08:02:22 | Captured traceback:
2017-09-03 08:02:22 | ~~~~~~~~~~~~~~~~~~~
2017-09-03 08:02:22 | Traceback (most recent call last):
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/ceilometer/tests/tempest/scenario/test_telemetry_integration.py", line 132, in test
2017-09-03 08:02:22 | self._do_test(filename)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/ceilometer/tests/tempest/scenario/test_telemetry_integration.py", line 125, in _do_test
2017-09-03 08:02:22 | self.fail(msg)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
2017-09-03 08:02:22 | raise self.failureException(msg)
2017-09-03 08:02:22 | AssertionError: From test "list servers grow" :
2017-09-03 08:02:22 | _StringException: Traceback (most recent call last):
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 93, in wrapper
2017-09-03 08:02:22 | func(self)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 141, in test_request
2017-09-03 08:02:22 | self._run_test()
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 463, in _run_test
2017-09-03 08:02:22 | raise failure
2017-09-03 08:02:22 | testtools.matchers._impl.MismatchError: 2 != 1: Unable to match $.servers.`len` as 2, got 1

2017-09-03 08:02:22 | tearDownClass (ceilometer.tests.tempest.scenario.test_telemetry_integration.TestTelemetryIntegration)
2017-09-03 08:02:22 | -----------------------------------------------------------------------------------------------------
2017-09-03 08:02:22 |
2017-09-03 08:02:22 | Captured traceback:
2017-09-03 08:02:22 | ~~~~~~~~~~~~~~~~~~~
2017-09-03 08:02:22 | Traceback (most recent call last):
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 190, in tearDownClass
2017-09-03 08:02:22 | six.reraise(etype, value, trace)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 173, in tearDownClass
2017-09-03 08:02:22 | teardown()
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 474, in clear_credentials
2017-09-03 08:02:22 | cls._creds_provider.clear_creds()
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 445, in clear_creds
2017-09-03 08:02:22 | self._clear_isolated_net_resources()
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 436, in _clear_isolated_net_resources
2017-09-03 08:02:22 | creds.subnet['name'])
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 386, in _clear_isolated_subnet
2017-09-03 08:02:22 | client.delete_subnet(subnet_id)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
2017-09-03 08:02:22 | return self.delete_resource(uri)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 41, in delete_resource
2017-09-03 08:02:22 | resp, body = self.delete(req_uri)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 301, in delete
2017-09-03 08:02:22 | return self.request('DELETE', url, extra_headers, headers, body)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
2017-09-03 08:02:22 | self._error_checker(resp, resp_body)
2017-09-03 08:02:22 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 780, in _error_checker
2017-09-03 08:02:22 | raise exceptions.Conflict(resp_body, resp=resp)
2017-09-03 08:02:22 | tempest.lib.exceptions.Conflict: An object with that identifier already exists
2017-09-03 08:02:22 | Details: {u'message': u'Unable to complete operation on subnet 54e1128c-849d-4242-9c6a-d2ec09c698f3: One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}
2017-09-03 08:02:22 |

[Arx Cruz (arxcruz)]

Telemetry must have these patches in order to work properly:

https://review.openstack.org/#/c/496723/
https://review.openstack.org/#/c/496800/
https://review.openstack.org/#/c/496694/
https://review.openstack.org/#/c/496751/
https://review.openstack.org/#/c/497835/

Checking [1] you can see that there's no panko in the event pipeline, which means the patch [2] isn't applied.

An example of telemetry working can be found in [3]

1 - http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario001-multinode-oooq/cb596ff/logs/subnode-2/etc/ceilometer/event_pipeline.yaml.txt.gz
2 - https://review.openstack.org/#/c/496723/
3 - http://logs.openstack.org/76/490376/16/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/0f84b43/logs/subnode-2/etc/ceilometer/event_pipeline.yaml.txt.gz

[Sagi (Sergey) Shnaidman (sshnaidm)]

@Arx, you're right, consistent repo is stuck on hash c9c4d7e and this is before those patches were merged. So seems like we need to disable tempest in this periodic job until we move forward with promotion..

[Emilien Macchi (emilienm)]

Sagi, before disabling tempest in promotion job, I would rather see with Prad or someone from Telemetry what we can do. If we don't get an answer by Monday EOD or if the fix takes too long, then ok to workaround.

[Arx Cruz (arxcruz)]

@Emilien
We don't need to disable tempest in promotion job, we just need to remove this test from the whitelist. The boot volume pattern still working as you can see http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario001-multinode-oooq/cb596ff/logs/tempest.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/500507

[OpenStack Infra (hudson-openstack)]

Change abandoned by Arx Cruz (<email address hidden>) on branch: master
Review: https://review.openstack.org/500507

[Sagi (Sergey) Shnaidman (sshnaidm)]

Agreed to wait until promotion on master, it will include required code.