Secure TLS configuration by default

[James Hebden]

From the discussion on bug 1851673, I am opening this bug to track modernisation of TLS configuration across all OpenStack charms.

It is important that the charms are providing opinionated, good practice defaults - we need to ensure TLS1.0 and TLS1.1 are removed from the supported ciphers as they are widely held to be deprecated and insecure. The current industry practice of disabling these TLS versions is described in the linked IETF draft: "Industry has actively followed guidance provided by NIST and the PCI Council to deprecate TLSv1.0 and TLSv1.1 by June 30, 2018. TLSv1.2 should remain a minimum baseline for TLS support at this time." [0]

In addition, we should downselect a good practice set of ciphers from the cross-section of ciphers with good client (browser, API client) support, and ciphers which follow current good cryptographic practice.

A good baseline for this comes from the Apache project documentation [1] which in turn is based on Mozilla's security guidance around server-side TLS Ciphers [2].

The below can be used to represent Mozilla's "intermediate" compatibility cipher suite, which enables TLS1.2 and TLS1.3. TLS1.2 is also important as I don't believe all shipped versions of apache2 with supported Ubuntu releases support TLS1.3 yet.

SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
SSLHonorCipherOrder on
SSLCompression off
SSLSessionTickets off

Users requiring insecure (or stronger!) cipher suites and versions would be able to customise them once 1851673 has been addressed, but until then, it would be simpler to give guidance around cipher support. For folks who need less secure configuration, we could simply give guidance around which specific charm releases support which TLS configurations. In my mind, having some users on older charm releases is better than blocking implementation of a more secure cipher suite on being able to customise the TLS configuration via charm settings.

[0] https://tools.ietf.org/id/draft-ietf-tls-oldversions-deprecate-02.html#rfc.section.2
[1] https://httpd.apache.org/docs/trunk/ssl/ssl_howto.html
[2] https://wiki.mozilla.org/Security/Server_Side_TLS

[James Hebden (ec0)]

Subscribing field high as this has become a security issue without a good workaround.

[Xav Paice (xavpaice)]

Just want to add a shout out for https://bugs.launchpad.net/nova/+bug/1842149 which adds configuration for nova-console, but we really want to have sane defaults without needing config.

[Nobuto Murata (nobuto)]

Sounds like a dup of https://bugs.launchpad.net/charm-helpers/+bug/1886630 ? I updated the cipher list as a part of 20.08.

[James Hebden (ec0)]

Thanks nobuto, I was not aware of that specific bug, and I'm glad we're all on the same page as to how this should be addressed :)

I'm glad that this has been potentially addressed in charmhelpers. I believe that all charms using apache2 will also need these fixes to address this, as this issue exists also outside of the dashboard. I'm glad that you and Yoshi have done that work. There is also the case of the nova-vnc-proxy which handles this, Xav has already mentioned this is soon to be configurable in nova itself, but this needs associated charm work to ship a good practice configuration as well, as I do not understand this to be using the same TLS-termination which is used in other charms.

Essentially, I believe a thorough review of all charms which expose TLS endpoints is needed to ensure that the Cipher list and version configuration meets current industry security good practice. Until this is done, these issues will continue to arise when charmed-OpenStack environments are scanned for security compliance.

[Nobuto Murata (nobuto)]

> I'm glad that this has been potentially addressed in charmhelpers. I believe that all charms using apache2 will also need these fixes to address this

My assumption was that all of the OpenStack API charms inherit the Apache2 template in charm-helpers, but reactive charms don't. So I've made another review against charm-layer-openstack.
https://bugs.launchpad.net/layer-openstack/+bug/1886630

OpenStack components which are not covered by those templates are:
- openstack dashboard as it's not an API endpoint, but WebUI
https://opendev.org/openstack/charm-openstack-dashboard/commit/e462df7401ac144dce2aeb36b4e563980410a920
- VNC proxy as you and Xav mentioned

Agreed that it's good to scan all OpenStack charms if there is anything missing.

[Liam Young (gnuoy)]

The charms have a security audit action, we could also update that to check the TLS configuration. The security action is run as part of the gate tests.

[Billy Olsen (billy-olsen)]

I believe all the the API bits are now configured to use TLS 1.2 by default with the 20.10 release of the charms, as the common code was in the reactive layers or in charmhelpers and Nobuto provided fixes for these.

I believe remains to be done is an investigation into:

VNC/Spice console proxy per Xav's comment
RabbitMQ server charm
OVN-Central/Chassis charm for review of TLS 1.2 in OVN

The security audit action is good to update, but that feels like an enhancement to consider outside of this bug.

[Drew Freiberger (afreiberger)]

Note that a backport of python-eventlet to support xenial consoles at TLSv1.2 is being SRU'd in lp#1904988.

[Billy Olsen (billy-olsen)]

This is included in all charms that use charm-helpers with commit https://github.com/juju/charm-helpers/commit/27d6ceb385e44a0610c1a6aba8e225368c4af384 and all openstack-layer based charms with https://opendev.org/openstack/charm-layer-openstack/commit/0a24820971a2b2d5ed55f2891f83bdede52ba538. Will mark fix-released.

[Billy Olsen (billy-olsen)]

rabbitmq-server charm will depend on the base version of erlang as well. Versions in Focal and newer require no change as the default erlang vm (23.0) only accepts tlsv1.2 and higher (groovy+ will only accept tlsv1.3 and higher).

Older version of rabbitmq-server will require setting the ssl_options.versions to tlsv1.2 and tlsv1.3.

[Frode Nordahl (fnordahl)]

> OVN-Central/Chassis charm for review of TLS 1.2 in OVN

The default behavior of the Open vSwitch clients and servers is to use the highest protocol version supported [0] and it has been this way since Open vSwitch v2.4.0 [1] which was released in 2014.

The default configuration does allow the use of TLSv1,TLSv1.1,TLSv1.2, so if the intention of this bug is to disallow protocol versions prior to TLSv1.2 that would translate into action necessary for the OVN charms.

0: http://manpages.ubuntu.com/manpages/focal/man1/ovsdb-server.1.html
1: https://github.com/openvswitch/ovs/commit/b56ea5d54e072105b398d26421f9a4578fa6e05b

[Frode Nordahl (fnordahl)]

https://patchwork.ozlabs.org/<email address hidden>/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/816970

[Frode Nordahl (fnordahl)]

>> OVN-Central/Chassis charm for review of TLS 1.2 in OVN
>
> The default behavior of the Open vSwitch clients and servers is to use the highest protocol version supported [0] and it has been this way since Open vSwitch v2.4.0 [1] which was released in 2014.
>
> The default configuration does allow the use of TLSv1,TLSv1.1,TLSv1.2, so if the intention of this bug is to disallow protocol versions prior to TLSv1.2 that would translate into action necessary for the OVN charms.
>
> 0: http://manpages.ubuntu.com/manpages/focal/man1/ovsdb-server.1.html
> 1: https://github.com/openvswitch/ovs/commit/b56ea5d54e072105b398d26421f9a4578fa6e05b

Just an update on the Open vSwitch part of this bug. While the above is true, and there is an outstanding issue of updating the Open vSwitch defaults and documentation, due to how the defaults are set up for the OpenSSL library in Ubuntu, Open vSwitch and OVN is in effect not affected by this.

The Ubuntu OpenSSL library configuration will make Open vSwitch and OVN only enable TLSv1.2 and TLSv1.3 as long as no configuration is provided for the SSL_Protocols and SSL_Ciphers options.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/817382

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/817382
Committed: https://opendev.org/openstack/charm-rabbitmq-server/commit/ece87ba8ca0cf1450457c98c71166ef0b9a517a0
Submitter: "Zuul (22348)"
Branch: master

commit ece87ba8ca0cf1450457c98c71166ef0b9a517a0
Author: James Page <email address hidden>
Date: Wed Nov 10 10:21:22 2021 +0000

    Restrict TLS versions

    TLS < 1.2 is considered insecure; where possible limit the versions
    of TLS to 1.2 or higher, enabling support for TLS 1.3 when the
    required erlang and rabbitmq versions are installed.

    Change-Id: Iec5ab60488986f8e332ff0e9a11895822a61c1ee
    Closes-Bug: 1892450
    Func-Test-PR: https://github.com/openstack-charmers/zaza-openstack-tests/pull/668