nova_api container in unhealthy state after undercloud upgrade

[Jose Luis Franco]

After upgrading the undercloud from Queens to Rocky, the nova_api container appears in unhealthy state:

c69e75f6f21a rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp14/openstack-nova-api:14.0-76 "kolla_start" 3 hours ago Up 3 hours (unhealthy) nova_api

When introspecting into the container we can see the following log regardig health:

            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 105,
                "Log": [
                    {
                        "Start": "2018-11-02T09:50:59.239386348-04:00",
                        "End": "2018-11-02T09:50:59.822371791-04:00",
                        "ExitCode": 1,
                        "Output": "awk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ngrep: /etc/httpd/conf.d/10-nova_metadata_wsgi.conf: No such file or directory\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ncurl: (6) Could not resolve host: ; Unknown error\n\n000 :0 0.000 seconds\n"
                    },
                    {
                        "Start": "2018-11-02T09:51:29.82254615-04:00",
                        "End": "2018-11-02T09:51:30.526381904-04:00",
                        "ExitCode": 1,
                        "Output": "awk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ngrep: /etc/httpd/conf.d/10-nova_metadata_wsgi.conf: No such file or directory\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ncurl: (6) Could not resolve host: ; Unknown error\n\n000 :0 0.000 seconds\n"
                    },
                    {
                        "Start": "2018-11-02T09:52:00.526739616-04:00",
                        "End": "2018-11-02T09:52:01.179131974-04:00",
                        "ExitCode": 1,
                        "Output": "awk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ngrep: /etc/httpd/conf.d/10-nova_metadata_wsgi.conf: No such file or directory\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\n\n000 :0 0.001 seconds\ncurl: (6) Could not resolve host: ; Unknown error\n"
                    },
                    {
                        "Start": "2018-11-02T09:52:31.17947248-04:00",
                        "End": "2018-11-02T09:52:31.492017608-04:00",
                        "ExitCode": 1,
                        "Output": "awk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ngrep: /etc/httpd/conf.d/10-nova_metadata_wsgi.conf: No such file or directory\nawk: fatal: cannot open file `/etc/httpd/conf.d/10-nova_metadata_wsgi.conf' for reading (No such file or directory)\ncurl: (6) Could not resolve host: ; Unknown error\n\n000 :0 0.000 seconds\n"
                    },

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/615204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/615204
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=de6f82008dfa3f1f0eb222dbd6152fc80c378c4f
Submitter: Zuul
Branch: master

commit de6f82008dfa3f1f0eb222dbd6152fc80c378c4f
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri Nov 2 15:52:59 2018 +0100

    Omit grep process in nova_api healthcheck.

    Currently, the nova_api healthcheck uses the output
    from the ps -ef command passing it to grep looking
    for the nova_metadata process. However, the way it's
    done currently would print out the grep commmand itself
    which would cause a false negative.

     ()[root@undercloud-0 /]# ps -ef | grep nova-metadata
     root 10979 8969 0 10:53 ? 00:00:00
     grep --color=auto nova-metadata

    This patch uses pgrep intead, which does not create a
    grep process that could match the string, plus it
    provides a clearer idea on what we are trying to match.

    Change-Id: Id493b3d0088e44c831b138b4526681694481d0eb
    Closes-Bug: #1801365

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/618524

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618524
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=fa7bceb5dad887a7c68ff6f234ac779dee760f6f
Submitter: Zuul
Branch: stable/rocky

commit fa7bceb5dad887a7c68ff6f234ac779dee760f6f
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri Nov 2 15:52:59 2018 +0100

    Omit grep process in nova_api healthcheck.

    Currently, the nova_api healthcheck uses the output
    from the ps -ef command passing it to grep looking
    for the nova_metadata process. However, the way it's
    done currently would print out the grep commmand itself
    which would cause a false negative.

     ()[root@undercloud-0 /]# ps -ef | grep nova-metadata
     root 10979 8969 0 10:53 ? 00:00:00
     grep --color=auto nova-metadata

    This patch uses pgrep intead, which does not create a
    grep process that could match the string, plus it
    provides a clearer idea on what we are trying to match.

    Change-Id: Id493b3d0088e44c831b138b4526681694481d0eb
    Closes-Bug: #1801365
    (cherry picked from commit de6f82008dfa3f1f0eb222dbd6152fc80c378c4f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.5.0 release.