Incorrect configuration in apache2 for internal service port

[Felipe Reyes]

An ironic based environment is misconfigured in the apache configuration, the ssl termination endpoint should be passing the requests to http://localhost:6365/ although it's passing it to http://localhost:6375/, the symptoms seen are:

$ openstack baremetal conductor list
SSL exception connecting to https://10.5.250.100:6385/v1/conductors: HTTPSConnectionPool(host='10.5.250.100', port=6385): Max retries exceeded with url: /v1/conductors (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))

This behavior is currentl reproduced in a lab environment, although this same issue is being seen at the gate[0]

Here is the list of relevant open ports:

```
root@juju-d974dd-zaza-cf05be3e3598-10:/var/log/juju# ss -tpln | grep -P '(6385|6375|6365)'
LISTEN 0 4096 0.0.0.0:6385 0.0.0.0:* users:(("haproxy",pid=136051,fd=7))
LISTEN 0 128 0.0.0.0:6365 0.0.0.0:* users:(("ironic-api",pid=132954,fd=8),("ironic-api",pid=132954,fd=7),("ironic-api",pid=132579,fd=7))
LISTEN 0 4096 *:6385 *:* users:(("haproxy",pid=136051,fd=8))
LISTEN 0 511 *:6375 *:* users:(("apache2",pid=235171,fd=8),("apache2",pid=235170,fd=8),("apache2",pid=235169,fd=8))
```

Here it's the content of /etc/apache2/sites-enabled/openstack_https_frontend.conf:

```
Listen 6375
<VirtualHost 10.5.3.161:6375>
    ServerName 10.5.250.100
    SSLEngine on

    # This section is based on Mozilla's recommendation
    # as the "intermediate" profile as of July 7th, 2020.
    # https://wiki.mozilla.org/Security/Server_Side_TLS
    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    SSLHonorCipherOrder off

    SSLCertificateFile /etc/apache2/ssl/ironic/cert_10.5.250.100
    SSLCertificateKeyFile /etc/apache2/ssl/ironic/key_10.5.250.100
    ProxyPass / http://localhost:6375/
    ProxyPassReverse / http://localhost:6375/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
    KeepAliveTimeout 75
    MaxKeepAliveRequests 1000
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>
```

Error in the gate:

```

2023-06-01 21:25:46.511409 | focal-medium | 2023-06-01 21:25:46 [INFO] raise exceptions.SSLError(msg)
2023-06-01 21:25:46.511437 | focal-medium | 2023-06-01 21:25:46 [INFO] keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://172.16.220.230:6385/v1/conductors: HTTPSConnectionPool(host='172.16.220.230', port=6385): Max retries exceeded with url: /v1/conductors (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
```

[0] https://openstack-ci-reports.ubuntu.com/artifacts/80d/884252/8/check/jammy-zed/80dbfa2/

[Edward Hope-Morley (hopem)]

same issue with octavia on zed/stable:

# egrep "ProxyPass|Listen" /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 9866
    ProxyPass / http://localhost:9866/
    ProxyPassReverse / http://localhost:9866/

i.e. it is proxying to itself. Changing those to 9865 fixes it.

[Edward Hope-Morley (hopem)]

This is the culprit - https://github.com/juju/charm-helpers/commit/6064a34627882d1c8acf74644c48d05db67ee3b4

If i remove that patch it works again.

[Edward Hope-Morley (hopem)]

The problem is that the above patch expects the following key to exist on the certificates relation:

octavia_1.processed_requests

But e.g. the octavia charm only has:

$ juju run --format json -u octavia/1 -- relation-get -r certificates:49 - vault/0| jq -r '.[].Stdout'| egrep "^\w+"
ca: |-
chain: |-
client.cert: |-
client.key: |-
egress-subnets: 10.5.2.221/32
ingress-address: 10.5.2.221
octavia_0.server.cert: |-
octavia_0.server.key: |-
octavia_1.server.cert: |-
octavia_1.server.key: |-
octavia_2.server.cert: |-
octavia_2.server.key: |-
private-address: 10.5.2.221

so it thinks it is not an https endpoint.

[Felipe Reyes (freyes)]

> i.e. it is proxying to itself. Changing those to 9865 fixes it.

yep, that's the behavior I observed. I'm rebuilding without the patch you pointed out, sadly I already destroyed my environment, so I can't check if the process_requests flag is set in my environment.

[Edward Hope-Morley (hopem)]

charm-helpers fix: https://github.com/juju/charm-helpers/pull/798

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/885518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/885518
Committed: https://opendev.org/openstack/charm-ironic-api/commit/6d7d772c2a6a0a936c87ad0ce4ca7ab8422d2561
Submitter: "Zuul (22348)"
Branch: master

commit 6d7d772c2a6a0a936c87ad0ce4ca7ab8422d2561
Author: Felipe Reyes <email address hidden>
Date: Wed Jun 7 11:43:23 2023 -0400

    Rebuild charm

    Closes-Bug: #2022980
    Change-Id: I3a69ad9f6725fef0a8ace253e793ea1aeea20bfa

[Edward Hope-Morley (hopem)]

now that https://review.opendev.org/c/openstack/charm-octavia/+/883868 is landed it will trigger a rebuild so we can mark octavia as done.

[Jadon Naas (jadonn)]

There was a backport to 2023.1 that was merged according to this review: https://review.opendev.org/c/openstack/charm-octavia/+/886091. I updated the status on this bug for charm-octavia for 2023.1