Cannot set bind-port=443 on TLS enabled swift-proxy application

[Soumya]

Hi,

I am trying to add swift using swift-proxy (version=94) and swift-storage (version=271) charms on an openstack cluster setup using juju charms.

My ha_bundle.yml for setting up swift is as follows:

———————————
Applications:
...
...
    swift-proxy:
    charm: cs:swift-proxy-94
    num_units: 3
    to:
    - lxd:0
    - lxd:1
    - lxd:2
    options:
      bind-port: 443
      harden: apache
      os-admin-hostname: swift-proxy.domain.com
      os-internal-hostname: swift-proxy.domain.com
      os-public-hostname: swift-proxy.domain.com
      replicas: 3
      ssl_cert: SSLCERT
      ssl_key: SSLKEY
      swift-hash: SWIFTHASH
      vip: VIP
      zone-assignment: auto
    swift-proxy-hacluster:
      charm: cs:hacluster-55
      options:
        cluster_count: 3
    swift-storage:
      charm: cs:swift-storage-271
      num_units: 3
      to:
      - "3"
      - "4"
      - "5"
      options:
        block-device: /dev/sda /dev/sdb
        encrypt: true
        overwrite: "true"
        zone: 1
...
...
relations:
...
...
- - ntp:juju-info
  - swift-storage:juju-info
- - swift-storage:secrets-storage
  - vault:secrets
- - swift-proxy:swift-storage
  - swift-storage:swift-storage
- - swift-proxy:identity-service
  - keystone:identity-service
- - swift-proxy-hacluster:ha
  - swift-proxy:ha
- - swift-proxy:object-store
  - glance:object-store
...
...
—————————————

juju deploy of the above bundle sets swift-proxy application status to 'blocked'. This happens when TLS is enabled for swift-proxy and bind-port is set to 443. Any other port works without any issues. In order to understand the reason, I did a step by step analysis.

My analysis:

- swift-proxy charm installs apache2. Without TLS, apache listens only on port 80. But when apache2 gets installed file "/etc/apache2/ports.conf" gets added which has code to Listen on ports 443 and 80.

——————————
$ juju ssh swift-proxy/22 sudo netstat -plane | egrep "(haproxy|apache2)" | grep tcp
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 0 3807263459 54698/haproxy
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 0 3807263457 54698/haproxy
tcp6 0 0 :::80 :::* LISTEN 0 2582379423 27931/apache2
tcp6 0 0 :::8080 :::* LISTEN 0 3807263460 54698/haproxy

$ juju ssh swift-proxy/22 sudo cat /etc/apache2/ports.conf
# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 80

<IfModule ssl_module>
 Listen 443
</IfModule>

<IfModule mod_gnutls.c>
 Listen 443
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
————————————

- SSL related modules get added to apache2 only when TLS is enabled (by setting ssl_cert and ssl_key config options in charm). After enabling TLS, apache starts listening on port 443 along with port 80.
————————————
$ juju ssh swift-proxy/22 sudo netstat -plane | egrep "(haproxy|apache2)" | grep tcp
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 0 860433626 6702/haproxy
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 0 860433624 6702/haproxy
tcp6 0 0 :::443 :::* LISTEN 0 860508554 31896/apache2
tcp6 0 0 :::8070 :::* LISTEN 0 860508560 31896/apache2
tcp6 0 0 :::80 :::* LISTEN 0 860508550 31896/apache2
tcp6 0 0 :::8080 :::* LISTEN 0 860433627 6702/haproxy
————————————

- Now when I set bind-port to 443 using (juju config swift-proxy bind-port=443), the units get into "blocked" state with reason "apache2 is not running". I assume this is because: charm updates haproxy with port 443. Then it tries to restart haproxy and apache2. haproxy starts successfully, but apache2 fails.

————————————
Unit Workload Agent Machine Public address Ports Message
swift-proxy/20* blocked idle 0/lxd/25 172.16.68.231 443/tcp,8080/tcp Services not running that should be: apache2
  swift-proxy-hacluster/144* active idle 172.16.68.231 Unit is ready and clustered
swift-proxy/21 blocked idle 9/lxd/23 172.16.68.232 443/tcp,8080/tcp Services not running that should be: apache2
  swift-proxy-hacluster/146 active idle 172.16.68.232 Unit is ready and clustered
swift-proxy/22 blocked idle 2/lxd/22 172.16.68.4 443/tcp,8080/tcp Services not running that should be: apache2
  swift-proxy-hacluster/145 active idle 172.16.68.4 Unit is ready and clustered
————————————

The workaround for this is to:

1. Fix apache configuration file by either:
** manually commenting port 443 related details in /etc/apache2/ports.conf or
** by commenting `Include ports.conf` and adding 'Listen 80' in /etc/apache2/apache.conf. Port 80 needs to be listening in units to fetch swift-ring related files (I used this approach).
2. Start the apache2 service.

After this, the units continues its execution and eventually application gets into an active state. Finally, apache starts listening on 433 (which I think is a special port used when bind-port=443).

————————————
$ juju ssh swift-proxy/22 sudo netstat -plane | egrep "(apache|haproxy)"
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 0 861442859 60846/haproxy
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 0 861442857 60846/haproxy
tcp6 0 0 :::443 :::* LISTEN 0 861442860 60846/haproxy
tcp6 0 0 :::433 :::* LISTEN 0 866467002 20581/apache2
————————————

I feel /etc/apache2/ports.conf should be managed by the charm to handle this scenario.

Note: I assume things will break when bind-port=80 is set on swift setup. I haven't tested it.

Thanks you!

[Meraki SRE Infrastructure Team (meraki-infra)]

Hello, bump on this.

[Meraki SRE Infrastructure Team (meraki-infra)]

We have to manually patch ports.conf every time we re-deploy our development cluster or deploy a new cluster, would be great to have this looked at please.