Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

[Sandeep Yadav]

Description:-

Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

Logs:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/e94c1bd/logs/undercloud/home/zuul/tempest/tempest.html.gz
~~~
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 88, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 404, in test_network_basic_ops
    self.check_public_network_connectivity(should_connect=True)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 204, in check_public_network_connectivity
    self.check_floating_ip_status(floating_ip, floatingip_status)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 855, in check_floating_ip_status
    st=status))
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': FloatingIP: {u'router_id': u'cbd91c04-5197-4f64-b1de-87fdf5a9eb98', u'status': u'DOWN', u'description': u'', u'tags': [], u'tenant_id': u'1d9e530cbe0649e69c1d6e992b0ff241', u'created_at': u'2020-11-11T16:56:27Z', u'updated_at': u'2020-11-11T16:56:27Z', u'floating_network_id': u'68e327db-1214-484c-8ab6-7cc00a5212e1', u'fixed_ip_address': u'10.100.0.2', u'floating_ip_address': u'192.168.24.116', u'revision_number': 0, u'project_id': u'1d9e530cbe0649e69c1d6e992b0ff241', u'port_id': u'e69ee175-6eea-4417-b404-d2a2c242fda3', u'id': u'bb96f5fd-d434-4177-a467-96559e77f231', u'qos_policy_id': None} is at status: DOWN. failed to reach status: ACTIVE
~~~

Another example:-

~~~
https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-queens/a69fbef/logs/undercloud/home/zuul/tempest/tempest.html.gz
~~~

[Marios Andreou (marios-b)]

Not sure this is related to tempest per-se *yet* anyway... the whole buildset was a bit of a disaster: https://review.rdoproject.org/zuul/buildset/e72ef803fabc4c8787af83c269b6a409

There were at least 3 different things failing there, including the issue reported in this bug.

=========
1. Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance c92fd45e-af51-4128-8a97-89c413d60aba., Code: 500

 https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/b1c346f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
        https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/0735299/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
 https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/b6dc652/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
 https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/72883bd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

=========
2. tempest Failed to establish authenticated ssh connection to cirros@192.168.24.115
        * https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens/3b78cdf/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz

=========
3. tempest issue reported in this bug... besides fs19 and fs18 already reported above, we had the same in * https://logserver.rdoproject.org/openstack-periodic-integration-stable6/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens/6fa9412/logs/undercloud/home/zuul/tempest.log.txt.gz

Ronelle already had a green run on fs18 with https://review.rdoproject.org/r/#/c/25325/
So this may all be 'infra issues' ... I am not sure we need bugs for these separate issues yet.

I posted https://review.rdoproject.org/r/#/c/30906/ for the rest of these missing jobs. If they still fail as above we can file the 2 new bugs.

[Ronelle Landy (rlandy)]

rerunning the failed tests on vexxhost:
https://review.rdoproject.org/r/#/c/25325/

[Marios Andreou (marios-b)]

well, definitely still a thing in my test run @ * https://review.rdoproject.org/r/#/c/30906/

for example

        * https://logserver.rdoproject.org/06/30906/1/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/b808132/logs/undercloud/home/zuul/tempest/tempest.html.gz
        * testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': FloatingIP: {u'router_id': u'e5da4051-cee8-4a0e-9657-da0afdbf49a7', u'status': u'DOWN', u'description': u'', u'tags': [], u'tenant_id': u'06d5c82de7d947a88ad07c5dc757a390', u'created_at': u'2020-11-13T11:54:48Z', u'updated_at': u'2020-11-13T11:54:48Z', u'floating_network_id': u'f1e464e0-c2d4-4135-87c2-c6fb5765b44b', u'fixed_ip_address': u'10.100.0.13', u'floating_ip_address': u'10.0.0.101', u'revision_number': 0, u'project_id': u'06d5c82de7d947a88ad07c5dc757a390', u'port_id': u'9f7eecad-e12e-4b9a-a437-976c066a3f03', u'id': u'07efa7c2-7c61-43a3-8a22-4bfac45c5db2', u'qos_policy_id': None} is at status: DOWN. failed to reach status: ACTIVE

TBD if this is 'rdo issues' because the vexx test from comment #2 isn't green yet either

[Ronelle Landy (rlandy)]

periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens - success today 11/18

[Ronelle Landy (rlandy)]

Moved fs19 and fs18 to a single playbook. May be fixed

[Ronelle Landy (rlandy)]

we have had two successful queens promotions - closing this out