test_load_balancer.LoadBalancerScenarioTest is failing on master - LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status updated to an invalid state of ERROR

[Ronelle Landy]

Closing 1895248 [master] (LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status updated to an invalid state of ERROR is failing on fs038 and opening a new bug in master below:

octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest is failing on master jobs:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_load_balancer.py", line 88, in test_load_balancer_ipv4_CRUD
    self._test_load_balancer_CRUD(4)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_load_balancer.py", line 127, in _test_load_balancer_CRUD
    CONF.load_balancer.lb_build_timeout)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/waiters.py", line 79, in wait_for_status
    raise exceptions.UnexpectedResponseCode(message)
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: (LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status updated to an invalid state of ERROR

The full error log is below:
https://b8a2a7cef1d4bf7fa218-26901191b574c0e7e1d178b146c22a89.ssl.cf2.rackcdn.com/810474/2/check/tripleo-ci-centos-8-scenario010-standalone/95e8cb1/logs/undercloud/var/log/tempest/stestr_results.html

Copied comments from #1895248 included below.

[Ronelle Landy (rlandy)]

From Michael

The above error from Wes is different than the first error where no image was loaded in glance. It should be a different launchpad bug as they are unrelated.
The error Wes linked to today is a nova API failure.

Octavia worker log reported:
2021-09-23 05:53:30.165 43 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: 9b195052-8294-4855-afca-f4b1591ef28c) into port aaa37bba-4518-4621-b953-21db7380a3a9.: octavia.common.exceptions.ComputeUnknownException: Unknown exception from the compute driver: Unknown Error (HTTP 504).

The TripleO HAProxy instance reported:
Sep 23 05:53:29 standalone haproxy[12]: Server nova_osapi/standalone.ctlplane.localdomain is DOWN, reason: Layer7 timeout, check duration: 10001ms. 0 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 23 05:53:29 standalone haproxy[12]: proxy nova_osapi has no server available!

The nova API log reports a traceback:
2021-09-23 05:53:31.148 11 ERROR nova.api.openstack.wsgi [req-a8f16fdc-0e92-4ff8-a414-8a40b2af5018 5fcf112ef52f4c88a39150602a71f492 8de52c88481246d7a2b8ac0dfc19f78d - default default] Unexpected exception in API method: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 66dcf42ffb3d4b97ba22ee6e46213bc4
2021-09-23 05:53:31.148 11 ERROR nova.api.openstack.wsgi Traceback (most recent call last):

This second job run failure is caused by nova failing in scenario 10.

[Ronelle Landy (rlandy)]

From Gregory Thiemonge (gthiemonge) wrote 15 hours ago:

related backtrace in the neutron logs around the same timestamp:

2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'server_uuid': '9b195052-8294-4855-afca-f4b1591ef28c', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'aaa37bba-4518-4621-b953-21db7380a3a9'}, {'server_uuid': '9b195052-8294-4855-afca-f4b1591ef28c', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'aaa37bba-4518-4621-b953-21db7380a3a9'}, {'server_uuid': '9b195052-8294-4855-afca-f4b1591ef28c', 'name': 'network-vif-deleted', 'tag': 'aaa37bba-4518-4621-b953-21db7380a3a9'}, {'name': 'network-changed', 'server_uuid': '08e1a465-e565-4194-aebf-71e6084ade4a', 'tag': '6fdbbf55-2947-487c-9a7b-dda678bbc70e'}]: novaclient.exceptions.ClientException: Unknown Error (HTTP 503)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova Traceback (most recent call last):
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/neutron/notifiers/nova.py", line 266, in send_events
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova batched_events)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/v2/server_external_events.py", line 39, in create
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova return_raw=True)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova resp, body = self.api.client.post(url, body=body)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/client.py", line 78, in request
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova raise exceptions.from_response(resp, body, url, method)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova novaclient.exceptions.ClientException: Unknown Error (HTTP 503)
2021-09-23 05:53:33.412 28 ERROR neutron.notifiers.nova

Michael Johnson (johnsom) wrote 15 hours ago: #5
This was discussed in the #tripleo channel last Friday here:
https://meetings.opendev.org/irclogs/%23tripleo/%23tripleo.2021-09-17.log.html#t2021-09-17T14:43:37

An in the #openstack-nova channel:
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-09-17.log.html#t2021-09-17T16:35:02

[Bogdan Dobrelya (bogdando)]

I believe the crux here is the same as https://bugs.launchpad.net/tripleo/+bug/1895248/comments/6