centos 8 security & tripleo components + integration pipeline master - Failed container(s): ['nova_wait_for_api_service

[Marios Andreou]

At [1][2] for security component centos8 standalone-on-multinode-ipa-security-master but also at [3] for master integration pipeline periodic centos-8 ovb featureset039 the deployment fails with trace like:

        2020-08-03 16:22:18.815903 | fa163e18-e9b1-70c8-fdf6-000000004622 | TIMING | tripleo_container_manage : Check podman create status | 0:44:00.711 | 610.41s
         [ERROR]: Container(s) which finished with wrong return code:
        ['nova_wait_for_api_service']
        2020-08-03 16:22:20.558282 | fa163e18-e9b1-70c8-fdf6-000000004624 | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

Looking at the stdouts can't quickly find some error [4].

This blocks the security component promotion [5] which is getting critical at ~2 weeks old.

This is also seen in periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master and blocking the tripleo component promotion [6]. See comment #3 below for pointers to logs for the tripleo component.

[1] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-master/745247c/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-master/9f986b2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/7e5b459/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[4] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-master/9f986b2/logs/undercloud/var/log/containers/stdouts/nova_wait_for_api_service.log.txt.gz
[5] https://github.com/rdo-infra/ci-config/blob/6a3a8d0f20c9733268f0c4524d144d796b3e2370/ci-scripts/dlrnapi_promoter/config/CentOS-8/component/master.yaml#L59
[6] https://github.com/rdo-infra/ci-config/blob/6c71fee284d225f463b68fbe066856b0ddb522c6/ci-scripts/dlrnapi_promoter/config/CentOS-8/component/master.yaml#L70

[Marios Andreou (marios-b)]

ongoing new logs there from 05th https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-master/b654aa2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

2020-08-05 16:14:23.873553 | fa163eec-dc55-4897-a722-0000000045f3 | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

[Marios Andreou (marios-b)]

this is an ongoing issue we need folks from DF and/or Security DFG to check logs. Due to other issues there were no runs on this pipeline yesterday [1] so there are no newer logs than comment #1 yet.

however it is definitely a blocker for the security component promotion which is quite stale now.

        * https://trunk.rdoproject.org/centos8/component/security/
        * promoted-components/ 2020-07-20 15:19
        * [DIR] current-tripleo/ 2020-07-20 15:19 -
        * [DIR] current/ 2020-08-05 19:33

[1] https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-security

[Marios Andreou (marios-b)]

just noticed that this is also seen for the tripleo component - updating the description and title to add this

        * https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master/c8dfa99/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
        * 2020-08-07 20:39:53.177672 | fa163e08-f56b-dd24-5867-0000000045dd | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

        * https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master/108b448/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
        * 2020-08-04 20:39:18.235595 | fa163e4d-2cd8-2ac3-5618-0000000045dc | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

        * https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master/c8dfa99/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
        * 2020-08-07 20:39:53.177672 | fa163e08-f56b-dd24-5867-0000000045dd | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

        * https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master/ba3c3e9/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
        * 2020-08-06 20:43:04.075816 | fa163e56-cd63-43c1-61e6-0000000045e7 | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

        * https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master/8741ef0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
        * 2020-08-08 20:40:34.660407 | fa163ef8-692f-b709-6c82-0000000045d9 | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

        * https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-master/d741209/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
        * 2020-08-09 20:43:36.958481 | fa163ec7-6b72-3097-d976-0000000045e1 | FATAL | Check containers status | standalone-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_api_service'], check logs in /var/log/containers/stdouts/"}

[yatin (yatinkarel)]

Issue is caused with https://review.opendev.org/#/c/723920/ and https://review.opendev.org/#/c/744526 should clear it.

[Marios Andreou (marios-b)]

cool thanks ykarel ... just checked at components and tripleo/security haven't run yet today let's see once we get that patch (merged 3 hours ago) through consistent->component-ci-testing and into those jobs.

[Marios Andreou (marios-b)]

OK, we had a green run for security component [1][2] - the tripleo component was skipped yesterday (unrelated NODE_FAILUREs in pipelines). I think it's done but holding on the fix-released until we have at least more than 1 verified green run.

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-master
[2] https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-master/2c4354b/

[Marios Andreou (marios-b)]

k we've now had 3 green [1] 2 for security one for tripleo. moving fix-released move back if you disagree