OS::Heat::SoftwareDeployment failed due SSL certificate verification error

[neil nie]

When SSL is enforced, the resource OS::Heat::SoftwareDeployment time_out and caused the stack creation failure.

#follow is the log
Aug 5 12:46:37 nnie-example-script-template-server-masn56hswa2n os-collect-config: /opt/stack/venvs/os-collect-config/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Aug 5 12:46:37 nnie-example-script-template-server-masn56hswa2n os-collect-config: InsecurePlatformWarning
Aug 5 12:46:37 nnie-example-script-template-server-masn56hswa2n os-collect-config: 2015-08-05 12:46:37.344 1106 WARNING os_collect_config.cfn [-] [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

In /etc/os-collec-config.conf, the https is set for cfn, like following.
[DEFAULT]
command = os-refresh-config

[cfn]
metadata_url = https://xx.xxx.xxx.xxx:8000/v1/
stack_name = nnie-example-script-template
secret_access_key = 42ee9b138aaa47ea92fec5034ae4eeda
access_key_id = fdba6c55f1944ac08873a0622523de2c
path = server.Metadata

Root cause: in python module requests, the default value for “verify” is set to True, which means to enforce the certificate check. But there is no certificate info in user_data, which is set by heat.

Question: is it expected that the certificate should be installed in VM beforehand? If so, heat needs to add certificate into user_data. Or is it reasonable to relax the certificate check for this type cases?

#source of heat/engine/resources/server.py
        elif self.transport_poll_server_cfn():
            meta['os-collect-config'] = {'cfn': {
                'metadata_url': '%s/v1/' % cfg.CONF.heat_metadata_server_url,
                'access_key_id': self.access_key,
                'secret_access_key': self.secret_key,
                'stack_name': self.stack.name,
                'path': '%s.Metadata' % self.name}
            }
#source of os_collect_config/cfn.py
            try:
                content = self._session.get(
                    url, params=params, headers=headers,
                    verify=CONF.cfn.ca_certificate)

Regards,
Neil

[Steve Baker (steve-stevebaker)]

This likely requires some coordination between heat and os-collect-config.

The absolute bare minimum would be for os-collect-config.conf [cfn] and [heat] to gain an 'insecure' option, and for heat to populate that from its own /etc/heat/heat.conf [clients_heat] insecure.

Beyond that, /etc/heat/heat.conf [clients_heat] also has options ca_file, cert_file, key_file. We would need security expert input on whether it is appropriate to populate boot user_data with the contents of these files to allow similar options to be set in os-collec-config.conf.

If this is appropriate then the heat user_data cloud-init items can include the contents of ca_file, cert_file, key_file and heat can also populate os-collect-config.conf with the resulting paths.

If this is not appropriate then os-collect-config will need to check for cert paths by convention, and the image building process needs to copy in those cert files.

[neil nie (neil-nie)]

Steve,

Thanks for the response. I did private change to relax the certificate check for "cfn" to work around this issue. To support other transport types, looks like more changes are required.

About how to add the certificate on VM, I agree we need security experts'comment. Another option is to add the certificate when creating nova server, which is similar to add SSH key.

Regards,
Neil

[Anant Patil (ananta)]

To start with, we atleast need the insecure option, which should be configurable from heat.conf in [cfn] and [heat]. The os-collect-config's conf should then be configured accordingly. A lot of tests are failing downstream simply because of this.

[Miika Kankare (kuula)]

Any good workarounds or options for this?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/285959

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/285962

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/285962
Reason:
stable/kilo closed for 2015.1.4

This release is now pending its final release and no freeze exception has
been seen for this changeset. Therefore, I am now abandoning this change.

If this is not correct, please urgently raise a thread on openstack-dev.

More details at: https://wiki.openstack.org/wiki/StableBranch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/360025

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/360025
Reason: lets do this here https://review.openstack.org/#/c/285959/

[Zane Bitter (zaneb)]

The proposed patch https://review.openstack.org/285959 lacks a way to get the *content* of the CA file to the booted server. The patches to o-c-c https://review.openstack.org/284725 and tripleo-image-elements https://review.openstack.org/285957 got abandoned for want of updates from the author (who no longer works on OpenStack).

All of this only affects the cfn collector, which is probably not the most interesting one anyway.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.opendev.org/285959
Reason: This is not close to a workable patch.