FS020 periodic train multinode job failed at following tempest tests:
http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/7e12b11/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
tempest.api.network.admin.test_agent_management.AgentManagementTestJSON.test_list_agent[id-9c80f04d-11f3-44a4-8738-ed2f879b0ff4]
--------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tempest/api/network/admin/test_agent_management.py", line 49, in test_list_agent
self.assertIn(self.agent, agents)
File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
self.assertThat(haystack, Contains(needle), message)
File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: {u'binary': u'ovn-controller', u'description': u'', u'admin_state_up': True, u'availability_zone': u'n/a', u'alive': False, u'topic': u'n/a', u'host': u'overcloud-novacompute-1.localdomain', u'agent_type': u'OVN Controller agent', u'id': u'f15d1f98-b0ce-4bf2-87b0-9ccd543b7d3e'} not in [{u'binary': u'ovn-controller', u'description': u'', u'admin_state_up': True, u'availability_zone': u'n/a', u'alive': True, u'topic': u'n/a', u'host': u'overcloud-novacompute-1.localdomain', u'agent_type': u'OVN Controller agent', u'id': u'f15d1f98-b0ce-4bf2-87b0-9ccd543b7d3e'}, {u'binary': u'networking-ovn-metadata-agent', u'description': u'', u'admin_state_up': True, u'availability_zone': u'n/a', u'alive': True, u'topic': u'n/a', u'host': u'overcloud-novacompute-1.localdomain', u'agent_type': u'OVN Metadata agent', u'id': u'906f7242-ac80-447d-8da4-95d47523e382'}, {u'binary': u'ovn-controller', u'description': u'', u'admin_state_up': True, u'availability_zone': u'n/a', u'alive': True, u'topic': u'n/a', u'host': u'overcloud-novacompute-0.localdomain', u'agent_type': u'OVN Controller agent', u'id': u'3ab6df89-1af2-479f-8ca3-6ed74dc897a7'}, {u'binary': u'networking-ovn-metadata-agent', u'description': u'', u'admin_state_up': True, u'availability_zone': u'n/a', u'alive': True, u'topic': u'n/a', u'host': u'overcloud-novacompute-0.localdomain', u'agent_type': u'OVN Metadata agent', u'id': u'208a64cf-7634-4428-9c14-696b69c064b0'}, {u'binary': u'ovn-controller', u'description': u'', u'admin_state_up': True, u'availability_zone': u'n/a', u'alive': True, u'topic': u'n/a', u'host': u'overcloud-controller-0.localdomain', u'agent_type': u'OVN Controller agent', u'id': u'40ef8c9b-d936-494c-9fce-1d16a6bb0d1b'}]
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
2019-12-11 01:19:35,607 307034 INFO [tempest.lib.common.rest_client] Request (AgentManagementTestJSON:test_list_agent): 200 GET http://10.0.0.5:9696/v2.0/agents 0.134s
2019-12-11 01:19:35,607 307034 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'status': '200', u'content-length': '2192', 'content-location': 'http://10.0.0.5:9696/v2.0/agents', u'date': 'Wed, 11 Dec 2019 01:19:35 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-e1e0c8a7-16e9-4cbb-8a9d-c6f8aa8b77ac'}
Body: {"agents": [{"binary": "ovn-controller", "description": "", "availability_zone": "n/a", "heartbeat_timestamp": "2019-12-11 01:19:35.531509", "admin_state_up": true, "alive": true, "topic": "n/a", "host": "overcloud-novacompute-1.localdomain", "agent_type": "OVN Controller agent", "id": "f15d1f98-b0ce-4bf2-87b0-9ccd543b7d3e", "configurations": {"chassis_name": "f15d1f98-b0ce-4bf2-87b0-9ccd543b7d3e", "bridge-mappings": "datacentre:br-ex"}}, {"binary": "networking-ovn-metadata-agent", "description": "", "availability_zone": "n/a", "heartbeat_timestamp": "2019-12-11 01:19:35.567526", "admin_state_up": true, "alive": true, "topic": "n/a", "host": "overcloud-novacompute-1.localdomain", "agent_type": "OVN Metadata agent", "id": "906f7242-ac80-447d-8da4-95d47523e382", "configurations": {"chassis_name": "f15d1f98-b0ce-4bf2-87b0-9ccd543b7d3e", "bridge-mappings": "datacentre:br-ex"}}, {"binary": "ovn-controller", "description": "", "availability_zone": "n/a", "heartbeat_timestamp": "2019-12-11 01:19:35.578088", "admin_state_up": true, "alive": true, "topic": "n/a", "host": "overcloud-novacompute-0.localdomain", "agent_type": "OVN Controller agent", "id": "3ab6df89-1af2-479f-8ca3-6ed74dc897a7", "configurations": {"chassis_name": "3ab6df89-1af2-479f-8ca3-6ed74dc897a7", "bridge-mappings": "datacentre:br-ex"}}, {"binary": "networking-ovn-metadata-agent", "description": "", "availability_zone": "n/a", "heartbeat_timestamp": "2019-12-11 01:19:35.586821", "admin_state_up": true, "alive": true, "topic": "n/a", "host": "overcloud-novacompute-0.localdomain", "agent_type": "OVN Metadata agent", "id": "208a64cf-7634-4428-9c14-696b69c064b0", "configurations": {"chassis_name": "3ab6df89-1af2-479f-8ca3-6ed74dc897a7", "bridge-mappings": "datacentre:br-ex"}}, {"binary": "ovn-controller", "description": "", "availability_zone": "n/a", "heartbeat_timestamp": "2019-12-11 01:19:35.600684", "admin_state_up": true, "alive": true, "topic": "n/a", "host": "overcloud-controller-0.localdomain", "agent_type": "OVN Controller agent", "id": "40ef8c9b-d936-494c-9fce-1d16a6bb0d1b", "configurations": {"chassis_name": "40ef8c9b-d936-494c-9fce-1d16a6bb0d1b", "bridge-mappings": "datacentre:br-ex"}}]}
It is failing continously from last 2 days.
http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/2553b48/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
we checked the neutron and ovn logs at all controller and compute nothing interesting found.
Looks like it might be related to this patch: https:/ /review. opendev. org/#/c/ 696936/