tripleo

fs001 and fs035 OVB jobs failing to set up private network for os_tempest

Bug #1984184 reported by Ronelle Landy on 2022-08-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo zed-1

Bug Description

fs001 and fs035 OVB jobs on master are failing to find the private network:

2022-08-08 08:09:30.888554 | primary | TASK [os_tempest : Ensure private network exists] ******************************
2022-08-08 08:09:30.888575 | primary | Monday 08 August 2022 08:09:30 -0400 (0:00:00.121) 1:19:49.866 *********
2022-08-08 08:09:32.689549 | primary | FAILED - RETRYING: Ensure private network exists (5 retries left).
2022-08-08 08:09:44.279397 | primary | FAILED - RETRYING: Ensure private network exists (4 retries left).
2022-08-08 08:09:55.853828 | primary | FAILED - RETRYING: Ensure private network exists (3 retries left).
2022-08-08 08:10:07.501238 | primary | FAILED - RETRYING: Ensure private network exists (2 retries left).
2022-08-08 08:10:19.097558 | primary | FAILED - RETRYING: Ensure private network exists (1 retries left).
2022-08-08 08:10:30.741137 | primary | fatal: [undercloud -> undercloud]: FAILED! => {"attempts": 5, "changed": false, "extra_data": {"data": null, "details": "503 Service Unavailable: No server is available to handle this request.", "response": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n"}, "msg": "HttpException: 503: Server Error for url: https://[2001:db8:fd00:1000::5]:13696/v2.0/networks?tenant_id=edc1924b084144b2b0c82348799d58ce, 503 Service Unavailable: No server is available to handle this request."}

Example logs:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/09d6423/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/9be8adf/job-output.txt

https://logserver.rdoproject.org/e3/e3e490137fec186d83c6463403a95e77ca13c303/openstack-periodic-integration-main/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/9437e52/job-output.txt

Example neutron log from fs035:

https://logserver.rdoproject.org/60/44460/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/b2eee76/logs/overcloud-controller-0/var/log/containers/neutron/server.log.txt.gz

Tried these jobs on vexxhost as well as ibm cloud. fs001 gets past this step on ibm cloud - fs035 consistently fails.

A fairly consistent failure since around 08/06:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master&pipeline=openstack-periodic-integration-main&skip=0

Tags:

Ronelle Landy (rlandy) on 2022-08-10

Changed in tripleo:
milestone:	none → zed-1
importance:	Undecided → Critical
status:	New → Triaged
tags:	added: promotion-blocker

Revision history for this message

Ronelle Landy (rlandy) wrote on 2022-08-10 (last edit on 2022-08-10):

Looking at https://logserver.rdoproject.org/60/44460/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/b2eee76/logs/overcloud-controller-0/var/log/containers/neutron/server.log.txt.gz: ...

2022-08-08 20:49:34.409 2 CRITICAL neutron [None req-6f114209-249d-4a46-beac-78017e32f446 - - - - - -] Unhandled error: Exception: Could not retrieve schema from tcp:[fd00:fd00:fd00:2000::142]:6642,tcp:[fd00:fd00:fd00:2000::3cd]:6642,tcp:[fd00:fd00:fd00:2000::5e]:6642

Comparing with report on https://bugzilla.redhat.com/show_bug.cgi?id=1556822:

/var/log/audit/audit.log.3:type=AVC msg=audit(1660097438.312:15866): avc: denied { mac_admin } for pid=93426 comm="setfiles" capability=33 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=capability2 permissive=0

Revision history for this message

Ronelle Landy (rlandy) wrote on 2022-08-10 (last edit on 2022-08-10):

<ykarel> rlandy, yes sure, doesn't look related to selinux though as per last comment
<ykarel> seems caused by https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836372
<ykarel> rlandy, https://review.opendev.org/c/openstack/tripleo-heat-templates/+/851450 could be the culprit

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-10: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790

Changed in tripleo:
status:	Triaged → In Progress

Revision history for this message

yatin (yatinkarel) wrote on 2022-08-11:

<< <ykarel> rlandy, https://review.opendev.org/c/openstack/tripleo-heat-templates/+/851450 could be the culprit
Actually https://review.opendev.org/c/openstack/tripleo-heat-templates/+/851450 caused it, but is a legit change. Terry proposed a fix https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790

Revision history for this message

Marios Andreou (marios-b) wrote on 2022-08-11 (last edit on 2022-08-12):

Adding missing context (I was about to post a test but there is already one from rlandy)

Testing the patch from Terry [1] with testproject at https://review.rdoproject.org/r/c/testproject/+/36254/148/.zuul.yaml

latest result is at [2] so we didn't confirm the fix yet (didn't get far enough in either job to execute the failing tempest setup step)

[1] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790
[2] https://review.rdoproject.org/r/c/testproject/+/36254/148#message-f299626059a85ce929aa59f39ea138122f84e6dd

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-12: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/bd8a82d42aca46b4cbfb06a3aad3894b76d3e762
Submitter: "Zuul (22348)"
Branch: master

commit bd8a82d42aca46b4cbfb06a3aad3894b76d3e762
Author: Terry Wilson <email address hidden>
Date: Wed Aug 10 16:11:23 2022 -0500

Fix ovsdb-server for IPv6 listening addresses

    Setting a Connection target to ptcp:6642 causes ovsdb-server to
    listen on 0.0.0.0, when for IPv6 we need to listen on [::].
    Add is_ipv6 condition to set listen address conditionally.

Closes-Bug: #1984184
Change-Id: I1d04eedeb7290408f612933427a763288e4ba10b

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-15: Fix proposed to tripleo-heat-templates (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/853101

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-20: Fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/853101
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3e12309c54c8c45eb39805e9543101785e3bc339
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3e12309c54c8c45eb39805e9543101785e3bc339
Author: Terry Wilson <email address hidden>
Date: Wed Aug 10 16:11:23 2022 -0500

Fix ovsdb-server for IPv6 listening addresses

    Setting a Connection target to ptcp:6642 causes ovsdb-server to
    listen on 0.0.0.0, when for IPv6 we need to listen on [::].
    Add is_ipv6 condition to set listen address conditionally.

    Closes-Bug: #1984184
    Change-Id: I1d04eedeb7290408f612933427a763288e4ba10b
    (cherry picked from commit bd8a82d42aca46b4cbfb06a3aad3894b76d3e762)