ETCD server certificate incorrectly created

[Jose Maria Infanzon]

Brief Description
-----------------
ETCD endpoint is unreachable due to an erroneous ETCD server certificate.

Server cert seems to be missing keyEncipherment:

        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:kubernetes
            X509v3 Key Usage:
                Digital Signature, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1

Severity
--------
Minor

Steps to Reproduce
------------------
Run curl command to hit etcd listen client endpoint

curl --cert /etc/etcd/etcd-server.crt --key /etc/etcd/etcd-server.key -i https://<floating cluster address-ETCD_LISTEN_CLIENT_URLS>:2379/health

curl --cacert /etc/etcd/ca.crt --cert /etc/etcd/etcd-client.crt --key /etc/etcd/etcd-client.key https://<floating cluster address-ETCD_LISTEN_CLIENT_URLS>:2379/health

Expected Behavior
------------------
The secured endpoint should be accessible and health check result should be returned

Actual Behavior
----------------
Certificate error message is received:

Password:
curl: (60) Certificate key usage inadequate for attempted operation.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
All

Branch/Pull Time/Commit
-----------------------
N/A.

Last Pass
---------
N/A.

Timestamp/Logs
--------------
curl --cacert /etc/etcd/ca.crt --cert /etc/etcd/etcd-server.crt --key /etc/etcd/etcd-server.key https://192.168.206.1:2379/health

curl: (60) Certificate key usage inadequate for attempted operation.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Test Activity
-------------
N/A

Workaround
----------
Run curl command with -k

curl -k --cacert /etc/etcd/ca.crt --cert /etc/etcd/etcd-server.crt --key /etc/etcd/etcd-server.key https://192.168.206.1:2379/health

[Ghada Khalil (gkhalil)]

Screening: This doesn't appear to have an end user impact, but should be cleaned up in the stx master branch. It should be an easy fix.

[Ghada Khalil (gkhalil)]

Assigning to Zhipeng since he introduced this certificate in stx.5.0

[zhipeng liu (zhipengs)]

Here, the detail info already gives out the root cause
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.

It should not be an issue, you should use --cacert, and it works from your test result.

Thanks!
Zhipeng

[Greg Waines (greg-waines)]

Hey Zhipeng,

Jose did use --cacert and it did not work.

We believe we tracked it down to being an issue with /etc/etcd/etcd-server.crt
It appears to be missing 'Key Encipherment' in it's 'X509v3 extensions / X509v3 Key Usage' values.
Which you can see by doing:
    openssl x509 -in /etc/etcd/etcd-server.crt -noout -text

This seems to be preventing the Certificate from being properly validated.

For comparison you can look at the apiserver certificate
    openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text
It does have 'Key Encipherment' and can be validated fine.

e.g. see an upstream discussion on this https://bugzilla.redhat.com/show_bug.cgi?id=1579384

[Ghada Khalil (gkhalil)]

Hi Zhipeng, I've assigned this to Joao Portal in my team. We'll add you to the code review.

[zhipeng liu (zhipengs)]

OK, thanks! I will follow up it.

Zhipeng

[João Victor Portal (jvictorp)]

The problem occurs because the etcd server certificate doesn't have option "Key Encipherment" set.

Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/799903

[João Victor Portal (jvictorp)]

Test: generated a new image with this change (using dev branch), installed in an AIO-SX VM and executed curl command below.

controller-0:~$ sudo curl --cacert /etc/etcd/ca.crt --cert /etc/etcd/etcd-client.crt --key /etc/etcd/etcd-client.key -i https://192.168.206.1:2379/health
Password:
HTTP/1.1 200 OK
Date: Wed, 07 Jul 2021 15:05:41 GMT
Content-Length: 17
Content-Type: text/plain; charset=utf-8

{"health":"true"}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/799903
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/75a2537cc4684be3241aed4b5483f00bdcbb79c3
Submitter: "Zuul (22348)"
Branch: master

commit 75a2537cc4684be3241aed4b5483f00bdcbb79c3
Author: Joao Victor Portal <email address hidden>
Date: Wed Jul 7 16:11:32 2021 -0300

    Set 'Key Encipherment' in ETCD certificates

    The curl command to reach ETCD endpoint fails when keyEncipherment is
    not present in ETCD server certificate. The code added to playbook adds
    keyEncipherment to both server and client ETCD certificates. The
    keyEncipherment is not essential to client certificate, but it causes no
    problems.

    Closes-Bug: 1931719
    Signed-off-by: Joao Victor Portal <email address hidden>
    Change-Id: Iba120ec941a33de5a85dad024939f36d77bfa606