Wallaby - Upgrade with let's encrypt failed

[Vincent Lambert]

When i try to upgrade my installation from Victoria to Wallaby with the script or manually it failed because no matter what I do it installs the self-signed certificate and ignores let's encrypt certificate

my user_variables.yml for let's encrypt part:

haproxy_ssl: true
haproxy_ssl_letsencrypt_enable: True
haproxy_ssl_letsencrypt_install_method: "distro"
haproxy_ssl_letsencrypt_setup_extra_params: "--http-01-address {{ ansible_host }} --http-01-port 8888"
haproxy_ssl_letsencrypt_email: "redacted"

haproxy_extra_services:
  # an internal only service for acme-challenge whose backend is certbot running on any haproxy instance
  - service:
      haproxy_service_name: letsencrypt
      haproxy_backend_nodes: "{{ groups['haproxy_all'] }}"
      backend_rise: 1 #rise quickly to detect certbot running without delay
      backend_fall: 2
      haproxy_bind:
        - 127.0.0.1 #bind to the localhost as the host internal IP will be used by certbot
      haproxy_port: 8888
      haproxy_balance_type: http

[Olaf Herman (olafher)]

I have the same problem, caused by https://opendev.org/openstack/openstack-ansible-haproxy_server/commit/f14ba917986c28023fcc7673573d66774b9e29bb

[Olaf Herman (olafher)]

The final Let's Encrypt certificate chain is copied to /etc/ssl/private (see https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/tasks/haproxy_ssl_letsencrypt.yml#L106), a directory no longer being used as it was replaced by /etc/haproxy/ssl.

A hotfix for the certificate not working is to change `dest` for `Create new pem file for haproxy` in (haproxy_server role, usually in /etc/ansible/roles/haproxy_server)/tasks/haproxy_ssl_letsencrypt.yml to `"{{ haproxy_ssl_cert_path }}/haproxy_{{ ansible_facts['hostname'] }}-{{ haproxy_bind_external_lb_vip_address }}.pem"`
And then of course run `openstack-ansible /opt/openstack-ansible/playbooks/haproxy-install.yml`

Note that this hotfix will overwrite the self-signed and automatically generated certificate originally stored there.

[Dmitriy Rabotyagov (noonedeadpunk)]

Yes, I think it's valid fix. Except one thing, that self-signed should not be generated at all. I deployed sandbox and will run some tests to prove it's working

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/811985

[Jonathan Rosser (jrosser)]

I think that the self-signed is required before certbot has run for the first time, as haproxy won't start with an SSL configuration but missing certificate. HAProxy is 'in the datapath' for the first LE issuance so must be running before the first run of certbot.

Chicken/egg situation requires the self-signed cert to be available initially.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/811985
Committed: https://opendev.org/openstack/openstack-ansible-haproxy_server/commit/1195355b436dd497fd7318e7bb1e2d111a938256
Submitter: "Zuul (22348)"
Branch: master

commit 1195355b436dd497fd7318e7bb1e2d111a938256
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 17:47:49 2021 +0300

    Fix haproxy Let's Encrypt SSL path

    With releasing PKI role we broke Let's Encrypt option because of
    changing directories where certs should be located
    and not reflecting these changes for let's encrypt. At the same time
    we should not generate self-signed cert when let's encrypt path is used.

    Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/811742
    Closes-Bug: #1938961
    Change-Id: I1a6701b171782528373bc1d0a39e70e6d1ef20ab

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813945

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813945
Committed: https://opendev.org/openstack/openstack-ansible-haproxy_server/commit/00441b7108752683b0984c3ee427fb1d6c78c357
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 00441b7108752683b0984c3ee427fb1d6c78c357
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 17:47:49 2021 +0300

    Fix haproxy Let's Encrypt SSL path

    With releasing PKI role we broke Let's Encrypt option because of
    changing directories where certs should be located
    and not reflecting these changes for let's encrypt. At the same time
    we should not generate self-signed cert when let's encrypt path is used.

    Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/811742
    Closes-Bug: #1938961
    Change-Id: I1a6701b171782528373bc1d0a39e70e6d1ef20ab
    (cherry picked from commit 1195355b436dd497fd7318e7bb1e2d111a938256)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server xena-eom release.