fedora-28 standalone failing at neutron-haproxy-ovnmeta service

[Quique Llorente]

http://logs.openstack.org/66/652066/2/check/tripleo-ci-fedora-28-standalone/3132cc2/job-output.txt.gz
2019-04-16 09:15:08.057103 | primary | TASK [validate-services : Get failed containers for podman] ********************
2019-04-16 09:15:08.072763 | primary | Tuesday 16 April 2019 09:15:08 +0000 (0:00:00.808) 1:09:21.403 *********
2019-04-16 09:15:09.287342 | primary | changed: [undercloud]
2019-04-16 09:15:09.331782 | primary |
2019-04-16 09:15:09.332018 | primary | TASK [validate-services : Fail if we detect failed podman container] ***********
2019-04-16 09:15:09.348360 | primary | Tuesday 16 April 2019 09:15:09 +0000 (0:00:01.275) 1:09:22.678 *********
2019-04-16 09:15:09.512017 | primary | failed: [undercloud] (item=neutron-haproxy-ovnmeta-44b51029-7c6d-41e8-8d38-ce3d3e4b59e4 Exited (137) 4 minutes ago) => {
2019-04-16 09:15:09.512146 | primary | "changed": false,
2019-04-16 09:15:09.512349 | primary | "item": "neutron-haproxy-ovnmeta-44b51029-7c6d-41e8-8d38-ce3d3e4b59e4 Exited (137) 4 minutes ago"
2019-04-16 09:15:09.512389 | primary | }
2019-04-16 09:15:09.512420 | primary |
2019-04-16 09:15:09.512457 | primary | MSG:
2019-04-16 09:15:09.512487 | primary |
2019-04-16 09:15:09.512615 | primary | Failed container detected. Please check the following locations
2019-04-16 09:15:09.512703 | primary | /var/log/extras/failed_containers.log
2019-04-16 09:15:09.512778 | primary | /var/log/extras/podman

[Ronelle Landy (rlandy)]

Related change: https://review.openstack.org/653136 Disable validations on f28 standalone

[wes hayutin (weshayutin)]

https://review.openstack.org/#/c/653136/

[Cédric Jeanneret (cjeanner)]

hiding the failure isn't a fix.

[Quique Llorente (quiquell)]

Looks like this is happening as expected

12:04 <dalvarez> quiquell|rover: Tengu numans mistery solved :) https://github.com/openstack/neutron/blob/master/neutron/agent/linux/external_process.py#L98
12:04 <dalvarez> quiquell|rover: Tengu numans when haproxy sidecar container is no longer needed, the ovn metadata agent kills it, hence you see that 137. It's all good
12:04 <dalvarez> and expected
1

So we have two options:
1 - discard sidecars from service validation
2 - do service validation before tempest.

Also why is not happening at centos7 ?

[Daniel Alvarez (dalvarezs)]

I believe it's happening as well on centos7 but perhaps the 'podman' output is not showing the exit code? The way sidecar containers are spawned/killed in the OVN metadata agent hasn't changed and is not dependent on the OS.

[Quique Llorente (quiquell)]

validate_services is not set at centos-7 job [1] and by default the value is false [2]

[1] https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L7
[2] https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-standalone.yml#L45

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/653383

[Quique Llorente (quiquell)]

So looks like running tempest after validate-services make sense so we have reverse it here
https://review.openstack.org/653383, let's also re-activate f28 and c7 and see if it's working now.

[Quique Llorente (quiquell)]

Activaging validate_services for f28 and c7 standalone jobs here https://review.openstack.org/653390

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/653383
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=1c111804a8c91aa59f0796e7c1de0b1a21478e29
Submitter: Zuul
Branch: master

commit 1c111804a8c91aa59f0796e7c1de0b1a21478e29
Author: Quique Llorente <email address hidden>
Date: Wed Apr 17 12:37:09 2019 +0200

    Run validate-services before tempest

    Some of the actions done at tempest like create a vm instance restart
    some sidecar containers and they appear as failed containers since the
    purpose of validate_services is to check them after deploy running
    before tempest would be good enough.

    Change-Id: I750e4d2ffc139433f7ec2a9a5c4adc6467ad19c9
    Closes-Bug: #1824977