tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

[Sandeep Yadav]

Description:

tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer <> is immutable and cannot be updated.', 'debuginfo': None}

Some jobs failed:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_beb/760080/2/check/tripleo-ci-centos-8-scenario010-standalone/bebdad4/job-output.txt

Sometimes job timed out:-

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c76/758627/37/check/tripleo-ci-centos-8-scenario010-standalone/c767d77/job-output.txt

But same type of error is observed:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_beb/760080/2/check/tripleo-ci-centos-8-scenario010-standalone/bebdad4/logs/undercloud/var/log/tempest/tempest_run.log

~~~
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2020-10-28 18:24:52,826 214423 INFO [tempest.lib.common.rest_client] Request (HealthMonitorScenarioTest:test_LC_PING_healthmonitor_CRUD): 409 POST http://192.168.24.3:9876/v2.0/lbaas/pools 0.108s
    2020-10-28 18:24:52,826 214423 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: {"pool": {"name": "tempest-lb_member_pool1_hm-721095081", "loadbalancer_id": "e8e7e3d4-1fbe-4ad7-99e4-e2230da7954b", "lb_algorithm": "LEAST_CONNECTIONS", "protocol": "TCP"}}
        Response - Headers: {'date': 'Wed, 28 Oct 2020 18:24:52 GMT', 'server': 'Apache', 'content-length': '147', 'x-openstack-request-id': 'req-3f1986f5-f927-4f70-8d7e-d5a846993f89', 'connection': 'close', 'content-type': 'application/json', 'status': '409', 'content-location': 'http://192.168.24.3:9876/v2.0/lbaas/pools'}
            Body: b'{"faultcode": "Client", "faultstring": "Load Balancer e8e7e3d4-1fbe-4ad7-99e4-e2230da7954b is immutable and cannot be updated.", "debuginfo": null}'

{1} octavia_tempest_plugin.tests.scenario.v2.test_healthmonitor.HealthMonitorScenarioTest.test_LC_TCP_healthmonitor_CRUD [0.104099s] ... FAILED
~~~

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c76/758627/37/check/tripleo-ci-centos-8-scenario010-standalone/c767d77/logs/undercloud/var/log/tempest/tempest_run.log

~~~
Captured traceback-1:
~~~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/services/load_balancer/v2/base_client.py", line 406, in _cleanup_obj
        self.delete, uri)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 329, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 823, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: Conflict with state of target resource
    Details: {'faultcode': 'Client', 'faultstring': 'Load Balancer b24c7c20-87c7-4b31-b489-40a49165e503 is immutable and cannot be updated.', 'debuginfo': None}
~~~

[Sandeep Yadav (sandeepyadav93)]

Hello,

Checked with octavia team: gthiemonge

They have fixed a similar issue in octavia-tempest-plugin 2 week ago: https://review.opendev.org/#/c/755777/

But looks like dlrn is not building latest package for python-octavia-tests-tempest..

https://trunk.rdoproject.org/centos8-master/component/octavia/consistent/ have python-octavia-tests-tempest-1.4.1-0.20201020083552.a80a185.el8.src.rpm have hash of patch(a80a185) merged in sept 17 - https://github.com/openstack/octavia-tempest-plugin/commits/master

Looks like we need https://review.opendev.org/#/c/755777/ hash 3bc1f4d (merged 6 days ago)to be in to fix this issue.

[Sandeep Yadav (sandeepyadav93)]

Hi,

Checked with #rdo team

octavia-tempest-plugin is pinned due to some missing httpx deps https://review.rdoproject.org/etherpad/p/httpx-centos8-octavia

However, octavia-tempest-plugin is pinned from long, so possible that some other component has caused the issue.

[Adriano Petrich (apetrich)]

could that be related to this? https://review.opendev.org/#/c/759280/

[Marios Andreou (marios-b)]

i've taken over as ruck this week so catching up here

For upstream check/gate, the job is non voting so this isn't blocking.

For the integration pipeline, this is not a promotion blocker (at least not yet). The job *is* in the promotion criteria [1] but it also looks to be running green in the integration pipeline [2].

I'm going to remove the promotion blocker tag for now, please re-add if you disagree thanks.

[1] http://38.145.34.55/config/CentOS-8/master.ini
[2] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-master

[wes hayutin (weshayutin)]

The bug may be blocking network and octavia

http://dashboard-ci.tripleo.org/d/M1DYSidGz/component-pipeline?orgId=1&var-launchpad_tags=alert&var-releases=master&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-influxdb_filter=job_name%7C%3D~%7C%2Fperiodic-tripleo-ci-centos-8-scenario010-standalone-.*%2F&from=now-30d&to=now&fullscreen&panelId=429

[wes hayutin (weshayutin)]

/me notes this job is NON-VOTING, but totally RED in master

http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&from=now-7d&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-8-scenario010-standalone&var-influxdb_filter=branch%7C%3D%7Cmaster

[Marios Andreou (marios-b)]

per comment #5 I checked the top three fails in the network component and that is a different issue - failing during deployment there so no that one is not related.

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/284fed3/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/9edc2a3/job-output.txt
https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/a65bf71/job-output.txt