fs001 and fs035 OVB jobs failing to set up private network for os_tempest
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Unassigned |
Bug Description
fs001 and fs035 OVB jobs on master are failing to find the private network:
2022-08-08 08:09:30.888554 | primary | TASK [os_tempest : Ensure private network exists] *******
2022-08-08 08:09:30.888575 | primary | Monday 08 August 2022 08:09:30 -0400 (0:00:00.121) 1:19:49.866 *********
2022-08-08 08:09:32.689549 | primary | FAILED - RETRYING: Ensure private network exists (5 retries left).
2022-08-08 08:09:44.279397 | primary | FAILED - RETRYING: Ensure private network exists (4 retries left).
2022-08-08 08:09:55.853828 | primary | FAILED - RETRYING: Ensure private network exists (3 retries left).
2022-08-08 08:10:07.501238 | primary | FAILED - RETRYING: Ensure private network exists (2 retries left).
2022-08-08 08:10:19.097558 | primary | FAILED - RETRYING: Ensure private network exists (1 retries left).
2022-08-08 08:10:30.741137 | primary | fatal: [undercloud -> undercloud]: FAILED! => {"attempts": 5, "changed": false, "extra_data": {"data": null, "details": "503 Service Unavailable: No server is available to handle this request.", "response": "<html>
Example logs:
Example neutron log from fs035:
Tried these jobs on vexxhost as well as ibm cloud. fs001 gets past this step on ibm cloud - fs035 consistently fails.
A fairly consistent failure since around 08/06:
Changed in tripleo: | |
milestone: | none → zed-1 |
importance: | Undecided → Critical |
status: | New → Triaged |
tags: | added: promotion-blocker |
Looking at https:/ /logserver. rdoproject. org/60/ 44460/2/ check/periodic- tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t035-master/ b2eee76/ logs/overcloud- controller- 0/var/log/ containers/ neutron/ server. log.txt. gz: ...
2022-08-08 20:49:34.409 2 CRITICAL neutron [None req-6f114209- 249d-4a46- beac-78017e32f4 46 - - - - - -] Unhandled error: Exception: Could not retrieve schema from tcp:[fd00: fd00:fd00: 2000::142] :6642,tcp: [fd00:fd00: fd00:2000: :3cd]:6642, tcp:[fd00: fd00:fd00: 2000::5e] :6642
Comparing with report on https:/ /bugzilla. redhat. com/show_ bug.cgi? id=1556822:
/var/log/ audit/audit. log.3:type= AVC msg=audit( 1660097438. 312:15866) : avc: denied { mac_admin } for pid=93426 comm="setfiles" capability=33 scontext= unconfined_ u:unconfined_ r:unconfined_ t:s0-s0: c0.c1023 tcontext= unconfined_ u:unconfined_ r:unconfined_ t:s0-s0: c0.c1023 tclass=capability2 permissive=0