VirtualHost is not being set correctly in "openstack_https_frontend.conf" file when deploying hacluster application with bindings

[Tytus Kurek]

The issue was originally reported in bug 1735421, but for some reason the fix for designate / charms.openstack didn't land.

There is a Juju environment with the following spaces:

$ juju spaces
Space Subnets
clo 100.86.0.0/20
oam 100.107.0.0/22

The designate application has been deployed from the "master" branch in the upstream with the following settings:

$ juju deploy \
--bind "oam admin=clo dns-backend=clo internal=clo public=clo shared-db=clo" \
--config "./designate.yaml" \
--num-units 3 \
--series trusty \
--to 22/lxd/10,23/lxd/10,24/lxd/7 \
./designate \
designate

$ cat designate.yaml
designate:
  debug: "true"
  enable-host-header: "true"
  nameservers: "ns1.example.com. ns1.example.com. ns3.example.com."
  openstack-origin: "cloud:trusty-mitaka"
  use-syslog: "true"
  verbose: "true"
  vip: "100.86.0.11"
  worker-multiplier: 0.5

and designate-hacluster application has been deployed from charm store with the following settings:

$ juju deploy \
--bind "oam ha=clo" \
--config "./designate-hacluster.yaml" \
--series trusty \
./hacluster \
designate-hacluster

$ cat designate-hacluster.yaml
designate-hacluster:
  corosync_transport: "unicast"
  cluster_count: "3"

The resulting configuration files look as follows:

# cat /etc/haproxy/haproxy.cfg
global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 9000
    timeout connect 9000
    timeout client 90000
    timeout server 90000

listen stats
    bind 127.0.0.1:8888
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth admin:Sx2Ten1UQQL5BbjF6lfFJyNrBTbW6B2c

frontend tcp-in_designate-api_admin
    bind *:9001
    acl net_100.86.0.180 dst 100.86.0.180/255.255.240.0
    use_backend designate-api_admin_100.86.0.180 if net_100.86.0.180
    acl net_100.107.2.206 dst 100.107.2.206/255.255.252.0
    use_backend designate-api_admin_100.107.2.206 if net_100.107.2.206
    default_backend designate-api_admin_100.107.2.206

backend designate-api_admin_100.86.0.180
    balance leastconn
    server designate-0 100.86.0.180:8991 check
    server designate-1 100.86.0.179:8991 check
    server designate-2 100.86.0.148:8991 check

backend designate-api_admin_100.107.2.206
    balance leastconn
    server designate-0 100.107.2.206:8991 check
    server designate-1 100.107.2.198:8991 check
    server designate-2 100.107.2.174:8991 check

# cat /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 8991
<VirtualHost 100.107.2.206:8991>
    ServerName 100.86.0.11
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/designate/cert_100.86.0.11
    SSLCertificateKeyFile /etc/apache2/ssl/designate/key_100.86.0.11
    ProxyPass / http://localhost:8981/
    ProxyPassReverse / http://localhost:8981/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

and the following query results as follows:

$ curl --insecure https://100.86.0.11:9001
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

Setting os-* options in designate charm:

$ juju config designate os-admin-network="100.86.0.0/20"
$ juju config designate os-internal-network="100.86.0.0/20"
$ juju config designate os-public-network="100.86.0.0/20"

solves the problem:

# cat /etc/haproxy/haproxy.cfg
global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 9000
    timeout connect 9000
    timeout client 90000
    timeout server 90000

listen stats
    bind 127.0.0.1:8888
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth admin:Sx2Ten1UQQL5BbjF6lfFJyNrBTbW6B2c

frontend tcp-in_designate-api_admin
    bind *:9001
    acl net_100.86.0.180 dst 100.86.0.180/255.255.240.0
    use_backend designate-api_admin_100.86.0.180 if net_100.86.0.180
    acl net_100.107.2.206 dst 100.107.2.206/255.255.252.0
    use_backend designate-api_admin_100.107.2.206 if net_100.107.2.206
    default_backend designate-api_admin_100.107.2.206

backend designate-api_admin_100.86.0.180
    balance leastconn
    server designate-0 100.86.0.180:8991 check
    server designate-1 100.86.0.179:8991 check
    server designate-2 100.86.0.148:8991 check

backend designate-api_admin_100.107.2.206
    balance leastconn
    server designate-0 100.107.2.206:8991 check
    server designate-1 100.107.2.198:8991 check
    server designate-2 100.107.2.174:8991 check

# cat /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 8991
<VirtualHost 100.86.0.180:8991>
    ServerName 100.86.0.11
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/designate/cert_100.86.0.11
    SSLCertificateKeyFile /etc/apache2/ssl/designate/key_100.86.0.11
    ProxyPass / http://localhost:8981/
    ProxyPassReverse / http://localhost:8981/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

$ curl --insecure https://100.86.0.11:9001
{
  "versions": {
    "values": [
      {
        "id": "v1",
        "links": [
          {
            "href": "https://100.86.0.11:9001/v1",
            "rel": "self"
          }
        ],
        "status": "DEPRECATED"
      },
      {
        "id": "v2",
        "links": [
          {
            "href": "https://100.86.0.11:9001/v2",
            "rel": "self"
          }
        ],
        "status": "CURRENT"
      }
    ]
  }
}

It looks like the root cause is the IP address in "VirtualHost" definition.

Attached are log files from all designate units.

[Tytus Kurek (tkurek)]

As I said, it looks like the root cause is the IP address in "VirtualHost" definition, but there are additional questions which should be answered here:
- Why are there two backends while the binding was used?
- Why is the "designate-api_admin_100.107.2.206" backend the default backend while the binding was used?

[James Page (james-page)]

Well the issue fixed under bug 1735421 is resolved (haproxy not using network space bindings) - AFAICT you're describing a different issue here in that the Apache VirtualHost used with https is enabled is not binding to the correct network address unless configuration options are set.

Does this only effect reactive charms like designate, or do you see this across all charm types?

[Tytus Kurek (tkurek)]

James,

You're right - it's a different bug (just the symptoms are the same which is misleading, especially if you're in rush), however, please take a look on my questions regarding bindings in #2.

It's hard to say what's the scope. In the environment from which I provided the output and logs for designate, other "hacluster" applications were deployed without the binding.

There's another environment where "hacluster" applications were deployed with the binding. It is based on an old charm release (probably 17.02), however, and os-*-network settings were used to workaround bug 1735421 there. The "openstack_https_frontend.conf" file from glance/0 unit looks as follows:

# cat /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 9282
<VirtualHost 100.107.2.23:9282>
    ServerName 100.86.0.3
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/glance/cert_100.86.0.3
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/glance/cert_100.86.0.3
    SSLCertificateKeyFile /etc/apache2/ssl/glance/key_100.86.0.3
    ProxyPass / http://localhost:9272/
    ProxyPassReverse / http://localhost:9272/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<VirtualHost 100.86.0.112:9282>
    ServerName 100.86.0.3
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/glance/cert_100.86.0.3
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/glance/cert_100.86.0.3
    SSLCertificateKeyFile /etc/apache2/ssl/glance/key_100.86.0.3
    ProxyPass / http://localhost:9272/
    ProxyPassReverse / http://localhost:9272/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

[Tytus Kurek (tkurek)]

Output from designate unit in the second environment:

# cat /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 8991
<VirtualHost 100.107.2.78:8991>
    ServerName 100.86.0.11
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/designate/cert_100.86.0.11
    SSLCertificateKeyFile /etc/apache2/ssl/designate/key_100.86.0.11
    ProxyPass / http://localhost:8981/
    ProxyPassReverse / http://localhost:8981/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<VirtualHost 100.86.0.205:8991>
    ServerName 100.86.0.11
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/designate/cert_100.86.0.11
    SSLCertificateKeyFile /etc/apache2/ssl/designate/key_100.86.0.11
    ProxyPass / http://localhost:8981/
    ProxyPassReverse / http://localhost:8981/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/549134

[Nobuto Murata (nobuto)]

Cross-linking to https://bugs.launchpad.net/charms.openstack/+bug/1769973 since LP: #1769973 may be a dup of this one.

[Ryan Beisner (1chb1n)]

Patchset CI issues are resolved. The proposed change is ready for review.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549134
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=33fd6a52b5db48f641e591f98f2218852e0163ad
Submitter: Zuul
Branch: master

commit 33fd6a52b5db48f641e591f98f2218852e0163ad
Author: Tytus Kurek <email address hidden>
Date: Thu May 10 17:08:33 2018 +0200

    Set "VirtualHost" to reflect bindings

    This patchset fixes the issue with IP address not being set properly
    in "VirtualHost" section of "openstack_https_frontend.conf" file
    when deploying reactive charms in the HA mode.

    Change-Id: Iee9f9d3621b0055bc62eaaed16d21b8f85113af9
    Closes-Bug: 1745389