Set enable_proxy_headers_parsing = True when HAProxy is used

[Adrien Cunin]

[oslo_middleware]
enable_proxy_headers_parsing = True

should be set in the configuration of services that use oslo_middleware, when HAProxy is used.

Obvious example with Designate, currently a request on the public IP returns:

# curl --insecure https://PUBLIC_IP:9001
{
  "versions": {
    "values": [
      {
        "id": "v1",
        "links": [
          {
            "href": "http://PUBLIC_IP:9001/v1",
            "rel": "self"
          }
        ],
        "status": "DEPRECATED"
      },
      {
        "id": "v2",
        "links": [
          {
            "href": "http://PUBLIC_IP:9001/v2",
            "rel": "self"
          }
        ],
        "status": "CURRENT"
      }
    ]
  }
}

Notice http instead of https.

With the config change, https is returned as expected.

[Adrien Cunin (adri2000)]

One question is: can we enable that option even when there is HAProxy in front?

https://git.openstack.org/cgit/openstack/oslo.middleware/commit/?id=f62c3a74c07238d91efb17e9ac64373f08894490 says we shouldn't for security reasons.

https://bugs.launchpad.net/oslo.middleware/+bug/1590635 asks to change the default directly in oslo.middleware.

[Adrien Cunin (adri2000)]

See my comment https://bugs.launchpad.net/oslo.middleware/+bug/1590635. Let's wait a bit to see the outcome there.

[Shannon Mitchell (shannon-mitchell)]

We are still encountering this same issue with designate when paging over the public ssl endpoint. The link being sent back in the 'links' json var is coming back as 'http'(non-ssl) when it should be 'https'(ssl) over the public endpoint. This is due to it being behind a loadbalancer that is hitting the backend over http. Haproxy is configured to set the "X-Forwarded-Proto: https" header. Designate should be seeing this and setting the link reference to utilize https.

Looks like the wait was to see how the oslo_middleware ticket goes. It looks like they are keeping enable_proxy_headers_parsing in olso_middleware set to false for security reasons. We should probably be setting this to true if designate is being ran behind haproxy.

###########################################
# Haproxy config setting X-Forwarded-Proto
###########################################
root@loadbalancer1:~# grep designate_api-front-1 /etc/haproxy/haproxy.cfg -A7
frontend designate_api-front-1
    bind 10.0.2.150:9001 ssl crt /etc/ssl/private/haproxy.pem ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    option httplog
    option forwardfor except 127.0.0.0/8
    reqadd X-Forwarded-Proto:\ https
    mode http
    default_backend designate_api-back

###########################################
# Tcpdump proving the header is being sent.
###########################################
root@loadbalancer1:~# tcpdump -n -l -i br-mgmt port 9001 -A -s 65535 2>&1 | grep 'GET /v2/zones' -A10
.<......GET /v2/zones?limit=1 HTTP/1.1
Host: 10.0.2.150:9001
User-Agent: python-designateclient-2.9.0
X-Auth-Token: <removed>
Content-Type: application/json
Accept: application/json
X-Auth-All-Projects: True
X-Forwarded-Proto: https
X-Forwarded-For: 10.0.2.100
Connection: close

###########################################
# Return json data from the api call
# generating the tcpdump data above
###########################################
{
  "zones": [],
  "links": {
    "self": "http://10.0.2.150:9001/v2/zones?limit=1"
  },
  "metadata": {
    "total_count": 0
  }
}

#############
# Root Cause
#############

It looks like designate is using oslo_middleware and the enable_proxy_headers_parsing defaults to false.

https://docs.openstack.org/designate/latest/admin/samples/config.html

This should be set to true n the [oslo_middleware] section of designate.conf if designate is behind haproxy.

##################
# Workaround test
##################

cat <<EOF > /etc/openstack_deploy/user_designate_proxy_header_fix.yml
designate_designate_conf_overrides:
  oslo_middleware:
    enable_proxy_headers_parsing: true
EOF

cd /opt/openstack-ansible/playbooks
openstack-ansible os-designate-install.yml --tags designate-config

curl -s --insecure -X GET https://10.0.2.150:9001/v2/zones?limit=1 -H "User-Agent: python-designateclient-2.9.0" -H "X-Auth-Token: $XAUTHTOKEN" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-All-Projects: True" | python -m json.tool
{
    "links": {
        "self": "https://10.0.2.150:9001/v2/zones?limit=1"
    },
    "metadata": {
        "total_count": 0
    },
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/706090

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706090
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_designate/commit/?id=7cdb8ea27c017fd056edfea056114b5329f8248a
Submitter: Zuul
Branch: master

commit 7cdb8ea27c017fd056edfea056114b5329f8248a
Author: Shannon Mitchell <email address hidden>
Date: Wed Feb 5 13:09:28 2020 -0600

    Fix paging link protocol when behind haproxy

    When using the designate public endpoint through haproxy, designate
    is returning a paging link with the incorrect protocol. This change
    is to make sure designate configurs oslo_middleware to parse the
    X-Forwarded-Proto header set by haproxy.

    Change-Id: Ia3288ccfc655a2814c36204a5cb381d3aa57e53e
    Closes-Bug: #1713663

[masterpe (michiel-y)]

I see this bug also on Train, Can we also port this to other releases?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate xena-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_designate zed-eom release.