TLS Deployments don't work in Pike

[Donny Davis]

Pike deployments with TLS Templates do not actually work. It does not seem to matter if I load the TLS templates in the beginning, middle, end of a CLI deployment or using the UI and selecting the TLS check boxes.

The deployment is fully functional without the TLS templates.

Error thrown is here
Oct 28 17:05:15 localhost os-collect-config: "Error: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: change from {md5}1f337186b0e1ba5ee82760cb437fb810 to {md5}4076c547f0a3ac82cb029b56199f9f06 failed: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20171028-23582-1rj204q -c' returned 1: [ALERT] 300/130452 (42320) : parsing [/etc/haproxy/haproxy.cfg20171028-23582-1rj204q:28] : 'bind 192.168.60.10:13042' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.",

OS::TripleO::NodeTLSData is never loaded into heat as a resource.

This Tripleo install was upgraded from ocata to pike.

[Juan Antonio Osorio Robles (juan-osorio-robles)]

How did you try to do the deployment?

Did you include the private key in the TLS template? Note that besides passing the SSLCertificate you need to pass the SSLKey parameter; are you passsing it? What templates are you using for the CLI?

[Donny Davis (donny-g)]

I tried the deployment exactly how the docs say. I included my cert chain and ca in the appropriate templates. The CA file is loaded, and like i said in the bug report OS::TripleO::NodeTLSData is not loaded. This means it doesn't create the overcloud.pem file haproxy is looking for and things fail at step3

OS::TripleO::NodeTLSCAData loads just fine...

I my latest deployment that does not have TLS enabled ..... OS::TripleO::NodeTLSData is loaded into heat as a resource. It makes no sense

[Juan Antonio Osorio Robles (juan-osorio-robles)]

So, NodeTLSCAData merely puts the CA in the trusted list.

a very confusignly named NodeTLSData is what you're looking for, which is also created in heat by default, but it will be of type OS::Heat::None, which means it's a noop (no-operation).

Your enable-tls.yaml file (where you specify the certs the key and such), should look like this one https://github.com/openstack/tripleo-heat-templates/blob/master/environments/ssl/enable-tls.yaml

Please note that one part of it is overriding the NodeTLSData resource, which you can see here:

https://github.com/openstack/tripleo-heat-templates/blob/master/environments/ssl/enable-tls.yaml#L40

Do you have that in your template?

[Donny Davis (automatikdonn)]

Yes, I followed the docs... and this worked just fine in Ocata.

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Are you deploying without TLS and then with TLS? I think I found an issue on that workflow.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/517984

[Donny Davis (donny-g)]

Yes it deploys fine without TLS. I am not using an UPDATE to deploy TLS after the fact.

If I try to deploy with TLS using this guide
https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html

I am using an external LB at the moment because this feature doesn't work.

[Donny Davis (donny-g)]

I will give your patch a spin if that is helpful

[Donny Davis (donny-g)]

Also I am not using containers, I am using RPM based deployment

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Donny, could you provide a paste of how your enable-tls.yaml looks like (obfuscating the private key)? I was able to deploy with TLS in stable/pike without containers. I saw issues with the containerized deployment, but as you said, you're not using that.

[Donny Davis (donny-g)]

Sure

# *******************************************************************
# This file was created automatically by the sample environment
# generator. Developers should use `tox -e genconfig` to update it.
# Users are recommended to make changes to a copy of the file instead
# of the original, if any customizations are needed.
# *******************************************************************
# title: Enable SSL on OpenStack Public Endpoints
# description: |
# Use this environment to pass in certificates for SSL deployments.
# For these values to take effect, one of the tls-endpoints-*.yaml environments
# must also be used.
parameter_defaults:
  # The content of the SSL certificate (without Key) in PEM format.
  #Type: string
  SSLCertificate: |
    -----BEGIN CERTIFICATE-----
    A BUNCH OF KEY STUFF HERE
    -----END CERTIFICATE-----
  # The content of an SSL intermediate CA certificate in PEM format.
  #Type: string
  SSLIntermediateCertificate: ''
  # The content of the SSL Key in PEM format.
  # Mandatory. This parameter must be set by the user.
  #Type: string
  SSLKey: |
    -----BEGIN PRIVATE KEY-----
    A BUNCH OF PRIVATE KEY STUFF HERE
    -----END PRIVATE KEY-----
  # ******************************************************
  # Static parameters - these are values that must be
  # included in the environment but should not be changed.
  # ******************************************************
  # The filepath of the certificate as it will be stored in the controller.
  # Type: string
  #DeployedSSLCertificatePath: /etc/pki/tls/private/overcloud_endpoint.pem

  # *********************
  # End static parameters
  # *********************
resource_registry:
  OS::TripleO::NodeTLSData: ../../puppet/extraconfig/tls/tls-cert-inject.yaml

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Looks about right. Can you try using an absolute path for OS::TripleO::NodeTLSData instead of a relative one?

[Donny Davis (donny-g)]

I have tried every single possible way to utilize these templates, to include directly using the ones in /usr/share/openstack-tripleo-templates. That is why I opened the bug.

The result is the same.

Can you post the templates you used for a successful deployment.

Also I should have added my undercloud was upgraded from ocata to pike.

[Juan Antonio Osorio Robles (juan-osorio-robles)]

The deployment command looked like this https://paste.fedoraproject.org/paste/l4xDiFxCSvztguto7IAlkg

and my enable-tls.yaml looks like this: https://paste.fedoraproject.org/paste/9080At0~k8aed-qIelJqhg

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/517984
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d1f3b1f683216cf728846c6403994baefe02d5d6
Submitter: Zuul
Branch: master

commit d1f3b1f683216cf728846c6403994baefe02d5d6
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Nov 6 14:58:52 2017 +0200

    Remove certificate before updating it

    Containerized HAProxy always tries to load the SSL certificate; if TLS
    is not enabled it will create the file as a directory. This messes up
    with the script that actually injects the HAProxy certificate into the
    undercloud. To address this, we update that script to take this into
    account.

    Change-Id: Ifc748648cc0f8caaf5a551fd0bc5724b94f3087d
    Closes-Bug: #1728267

[Donny Davis (donny-g)]

So NODETLSData works fine for single node deployments as you noted.

However when doing HA with
/usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml

It does not work. Is this a puppet-pacemaker issue?

[Juan Antonio Osorio Robles (juan-osorio-robles)]

I haven't been able to reprodce the issue, not even in a 3 controller deployment. So I haven't been able to pinpoint what the actual issue is :/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b2 development milestone.

[James Bagwell (jimbagwell)]

Hello, I am encountering the same exact issue. Is there an actual work around? Please advise.

[Juan Antonio Osorio Robles (juan-osorio-robles)]

James, for the record, can you post the version of tripleo-heat-templates you have?

[Juan Antonio Osorio Robles (juan-osorio-robles)]

And the python-tripleoclient version

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/528851

[James Bagwell (jimbagwell)]

openstack-tripleo-heat-templates-7.0.1-0.20170925090156.06aa582.el7.centos.noarch
python-tripleoclient-7.3.1-0.20170922015906.1023935.el7.centos.noarch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/528851
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ef7da3e341da530ad85775359f675d3eeb5e52c9
Submitter: Zuul
Branch: stable/pike

commit ef7da3e341da530ad85775359f675d3eeb5e52c9
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Nov 6 14:58:52 2017 +0200

    Remove certificate before updating it

    Containerized HAProxy always tries to load the SSL certificate; if TLS
    is not enabled it will create the file as a directory. This messes up
    with the script that actually injects the HAProxy certificate into the
    undercloud. To address this, we update that script to take this into
    account.

    Change-Id: Ifc748648cc0f8caaf5a551fd0bc5724b94f3087d
    Closes-Bug: #1728267
    (cherry picked from commit d1f3b1f683216cf728846c6403994baefe02d5d6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.7 release.