horizon container error at start

[joek-office]

Hello together,
i have a problem with a plain kolla-ansible deployment. After deployment the horizon container is crashed with the following error message:

Running command: '/usr/sbin/httpd -DFOREGROUND'
+ exec /usr/sbin/httpd -DFOREGROUND
AH00526: Syntax error on line 85 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty

Using the container images from quay.io and kolla-ansible version 15.1.0. Deploying to
openstack_release: "zed"
kolla_base_distro: "rocky"

with no ssl enabled.
Have the image exported and the path of the certificate looks as follows:
/horizon-image# ls -lah etc/pki/tls/certs/
total 8,0K
drwxr-xr-x 2 root root 4,0K Mär 15 12:52 .
drwxr-xr-x 5 root root 4,0K Mär 15 03:08 ..
lrwxrwxrwx 1 root root 49 Sep 20 16:35 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root 55 Sep 20 16:35 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
--w------- 1 root root 0 Mär 15 12:52 localhost.crt

i think is a problem with this certificate (certificate is empty and file rights seems crazy for me) or with a previous used image.

Can anyone help or advise. Should i give further information?

[joek-office (joek-office)]

Today a new try. Today all is running. Bug can be closed.

[JOONGKEE KWAK (kift25)]

i got troubled same problem with above issue.
how to solve it?

'docker logs horizon' command's output is below

+ echo 'Running command: '\''/usr/sbin/httpd -DFOREGROUND'\'''
Running command: '/usr/sbin/httpd -DFOREGROUND'
+ exec /usr/sbin/httpd -DFOREGROUND
AH00526: Syntax error on line 85 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty

[joek-office (joek-office)]

In my case, i have removed the images from the system. So docker daemon have to download the newest from quay.io. You can remove the image with
 - docker image rm [Image ID]
or you prune all the images so that every image have to download.
 - docker image prune --all -f

[Nate Bill (nbill)]

Updated horizon container and now have this issue

Output of docker inspect:
            "Image": "quay.io/openstack.kolla/horizon:zed-rocky-9",
            "Labels": {
                "build-date": "20230330",
                "io.buildah.version": "1.28.0",
                "kolla_version": "15.1.1",
                "maintainer": "Kolla Project (https://launchpad.net/kolla)",
                "name": "horizon"
            }

Error:
++ . /usr/local/bin/kolla_httpd_setup
++++ whoami
+++ [[ root == \r\o\o\t ]]
+++ [[ rocky =~ debian|ubuntu ]]
+++ rm -rf '/var/run/httpd/*' '/run/httpd/*' '/tmp/httpd*'
+++ [[ rocky =~ centos|rocky ]]
+++ [[ ! -e /etc/pki/tls/certs/localhost.crt ]]
+ echo 'Running command: '\''/usr/sbin/httpd -DFOREGROUND'\'''
Running command: '/usr/sbin/httpd -DFOREGROUND'
+ exec /usr/sbin/httpd -DFOREGROUND
AH00526: Syntax error on line 85 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty

[Antony Messerli (antonym)]

Ran into this as well on latest zed-rocky-9:

            "Labels": {
                "build-date": "20230405",
                "io.buildah.version": "1.28.0",
                "kolla_version": "15.1.1",
                "maintainer": "Kolla Project (https://launchpad.net/kolla)",
                "name": "horizon"
            }

[KangWoo Lee (koovis)]

same problem

+ echo 'Running command: '\''/usr/sbin/httpd -DFOREGROUND'\'''
+ exec /usr/sbin/httpd -DFOREGROUND
Running command: '/usr/sbin/httpd -DFOREGROUND'
AH00526: Syntax error on line 85 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty
[root@tp-control1 ~]#

            "Labels": {
                "build-date": "20230412",
                "io.buildah.version": "1.28.0",
                "kolla_version": "15.1.1",
                "maintainer": "Kolla Project (https://launchpad.net/kolla)",
                "name": "horizon"
            }

[joek-office (joek-office)]

Think I will reset the status of this report.
Seems not fixed and multiple persons affected.

[Loan Harrouin (loanh)]

Got the same issue here. Deploying the really basic all-in-on configuration as explain in the doc: https://docs.openstack.org/project-deploy-guide/kolla-ansible/zed/quickstart.html

The original issue is coming from here:
++++ whoami
+++ [[ root == \r\o\o\t ]]
+++ [[ rocky =~ debian|ubuntu ]]
+++ rm -rf /var/run/httpd/htcacheclean /run/httpd/htcacheclean '/tmp/httpd*'
+++ [[ rocky =~ centos|rocky ]]
+++ [[ ! -e /etc/pki/tls/certs/localhost.crt ]]
+++ /usr/libexec/httpd-ssl-gencerts
Could not write to /tmp/dhparams.pem. Check directory permissions.

What is weird is that when i run a container based on this image as root, i am able to run the script /usr/libexec/httpd-ssl-gencerts and generate the localhost.crt file without issue.

[Loan Harrouin (loanh)]

This error happen because the file /tmp/dhparams.pem already exist.
So when the script /usr/local/bin/kolla_httpd_setup trigger the script /usr/libexec/httpd-ssl-gencerts,
he try to generate again the file but fail:

sscg -q \
     --cert-file /etc/pki/tls/certs/localhost.crt \
     --cert-key-file /etc/pki/tls/private/localhost.key \
     --ca-file /etc/pki/tls/certs/localhost.crt \
     --dhparams-file /tmp/dhparams.pem \
     --lifetime 365 \
     --hostname $FQDN \
     --email root@$FQDN

Not really sure what this file is about and why it has already been generated.

So we can add the -f option to the sscg call or in /usr/local/bin/kolla_httpd_setup we add this part:
rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd* /tmp/dhparams.pem

(at the start of the kolla script there is this information:
    # NOTE(pbourke): httpd will not clean up after itself in some cases which
    # results in the container not being able to restart. (bug #1489676, 1557036))

[Loan Harrouin (loanh)]

It's working on master-rocky-9.
I can see that in this version they modify the script /usr/local/bin/kolla_httpd_setup and added a line to rm the problematic file:

    if [[ "${KOLLA_BASE_DISTRO}" =~ centos|rocky ]] && [[ ! -e /etc/pki/tls/certs/localhost.crt ]]; then
        rm -f /tmp/dhparams.pem
        /usr/libexec/httpd-ssl-gencerts
    fi

But it seem that the fix hasn't been backport to zed release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla/+/881236

[KangWoo Lee (koovis)]

It's not fixed in this version
image: quay.io/openstack.kolla/horizon:zed-rocky-9

            "Labels": {
                "build-date": "20230424",
                "io.buildah.version": "1.28.0",
                "kolla_version": "15.1.1",
                "maintainer": "Kolla Project (https://launchpad.net/kolla)",
                "name": "horizon"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 1329439186,
        "VirtualSize": 1329439186,
        "GraphDriver": {

++++ whoami
+++ [[ root == \r\o\o\t ]]
+++ [[ rocky =~ debian|ubuntu ]]
+++ rm -rf '/var/run/httpd/*' '/run/httpd/*' '/tmp/httpd*'
+++ [[ rocky =~ centos|rocky ]]
+++ [[ ! -e /etc/pki/tls/certs/localhost.crt ]]
+ echo 'Running command: '\''/usr/sbin/httpd -DFOREGROUND'\'''
+ exec /usr/sbin/httpd -DFOREGROUND
Running command: '/usr/sbin/httpd -DFOREGROUND'
AH00526: Syntax error on line 85 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty

-------------------------------

[Maksim Malchuk (mmalchuk)]

Fix for Zed is only proposed on Apr 21, not released yet. Please be patient.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/881467

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/881236
Committed: https://opendev.org/openstack/kolla/commit/003fff5dc1cc94bd8da0fd032e353ddafd0f233a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 003fff5dc1cc94bd8da0fd032e353ddafd0f233a
Author: Michal Nasiadka <email address hidden>
Date: Mon Mar 6 18:27:46 2023 +0100

    https: remove dhparams.pem before running gencerts

    On EL9 the file exists and script errors with wrong
    permissions to it.

    Closes-Bug: #2011716
    Change-Id: Ib32baa3208e6bfc5520ff8537193c9a4d6cbada7
    (cherry picked from commit 12d431e3994023ce735a2352554a99a5cc9431c9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/881467
Committed: https://opendev.org/openstack/kolla/commit/e590ec353841db16a96ad68f7ad5a13bbf7ee168
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit e590ec353841db16a96ad68f7ad5a13bbf7ee168
Author: Michal Nasiadka <email address hidden>
Date: Mon Mar 6 18:27:46 2023 +0100

    https: remove dhparams.pem before running gencerts

    On EL9 the file exists and script errors with wrong
    permissions to it.

    Closes-Bug: #2011716
    Change-Id: Ib32baa3208e6bfc5520ff8537193c9a4d6cbada7
    (cherry picked from commit 12d431e3994023ce735a2352554a99a5cc9431c9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 14.10.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 15.2.0 release.