Nova console and vault provides internal TLS certificate only

[Márton Kiss]

When nova cloud controller is used with Vault, and internal and public endpoints are different, the spice console is providing the public service endpoint name, but the SSL connections are providing the certificate for the internal service only. It is causing a security warning when the customer wants to use the console from the horizon dashboard (which is using the public api endpoints).

$ openstack catalog show compute -f json
{
  "endpoints": [
    {
      "id": "9999f8531b4c4fca84aaa7d2f3513f65",
      "interface": "internal",
      "region_id": "R1",
      "url": "https://nova-internal.<domain>:8774/v2.1",
      "region": "R1"
    },
    {
      "id": "99991ac800934dbebf4f303ac1eda7ca",
      "interface": "admin",
      "region_id": "R1",
      "url": "https://nova-internal.<domain>:8774/v2.1",
      "region": "R1"
    },
    {
      "id": "999944277d2f40fe97f1afe5bbf29669",
      "interface": "public",
      "region_id": "R1",
      "url": "https://nova.<domain>:8774/v2.1",
      "region": "R1"
    }
  ],
  "id": "9999c921e2a14ebfb67ef030cb4ef7a9",
  "name": "nova",
  "type": "compute"
}

$ openstack console url show canonical-test01 -f json
{
  "type": "novnc",
  "url": "https://nova.<domain>:6080/vnc_auto.html?path=%3Ftoken%9999999-5a92-4be4-b8f5-0e774add79a5"
}

The ssl connection however provides the certificate for the internal endpoint:
$ openssl s_client -connect nova.<domain>:6082
CONNECTED(00000005)
depth=0 CN = nova-internal.<domain>
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = nova-internal.<domain>
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = nova-internal.<domain>
   i:CN = Vault Root Certificate Authority (charm-pki-local)
---

How could we enforce spice to return the proper certificate for the connection?

[Márton Kiss (marton-kiss)]

A workaround for the issue:

1. get the key and certificates from /etc/apache2/ssl/nova/cert_nova.<domain> and /etc/apache2/ssl/nova/key_nova.<domain>
2. use the deprecated console-ssl- options:
juju config nova-cloud-controller \
    console-ssl-cert="$(base64 cert_nova.<domain>)" \
    console-ssl-key="$(base64 key_nova.<domain>)"

This will overwrite the nova.conf entries with the specified certificate and key:
[DEFAULT]
...
cert=/etc/nova/ssl/nova_cert.pem
key=/etc/nova/ssl/nova_key.pem

The internal / admin / public TLS endpoints will be still terminated by apache, so this change won't affect nova service rest interfaces, only the consoleauth will pick up the setting:

nova 1913397 0.0 0.0 296568 104316 ? Ss 12:44 0:03 /usr/bin/python3 /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-consoleauth.log

[Pedro Guimarães (pguimaraes)]

Marking this issue as field-high since it is a blocker for ongoing delivery and its work-around depend on a flag that is flagged to be removed.

We need an alternative solution on the charm code before console-ssl-* configs can go away

[Liam Young (gnuoy)]

Please can you provide sanitised bundle and overlays ? Also a juju crashdump if possible.

[Liam Young (gnuoy)]

I think the code that should be adding them is here https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/cert_utils.py#L122 It will be interesting to see if there are any "Skipping request" entries in the logs

[Andrew McLeod (admcleod)]

We are still waiting on bundle/overlays/logs - has there been any progress on getting these?

[James Page (james-page)]

Marking as incomplete as requested log files have not been provided.

[James Page (james-page)]

I suspect the proxy's are falling back to using:

my_ip = 10.246.114.45

cert=/etc/apache2/ssl/nova/cert_10.246.114.45
key=/etc/apache2/ssl/nova/key_10.246.114.45

my_ip is always bound to the internal network space binding (I think)

[James Page (james-page)]

the cert and key in comment #7 are generated using the internal binding, not the public binding.

Working on a fix.

[James Page (james-page)]

https://review.opendev.org/742375

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742375
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=f0095ffcbbfd3c84dc410713ea269ea2dd3ea879
Submitter: Zuul
Branch: master

commit f0095ffcbbfd3c84dc410713ea269ea2dd3ea879
Author: James Page <email address hidden>
Date: Wed Jul 22 10:10:15 2020 +0100

    Use public endpoint binding for console cert and key

    Ensure that the public endpoint binding is used to resolve the
    path to the SSL certificate and key files as the base access
    URL for console access is always via this binding.

    Add unit tests to cover the InstanceConsoleContext class.

    Change-Id: I27de9445d249b0d670543d250bd02f450764a10f
    Closes-Bug: 1871428