StarlingX

system certificate-install -m ssl (and docker_registry) do not support intermediateCA-signed certificates

Bug #1883300 reported by Greg Waines
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Andy

Bug Description

Brief Description
-----------------
An intermediateCA-signed server certificate typically has 2x certs and 1xkey.
The server certificate, the signing intermediateCA's certificate and the server's key.

When trying to install this certificate, the following error occurs.

[sysadmin@controller-0 wcp-111(keystone_admin)]$ system certificate-install -m ssl endpoint-certificate-interca-signed.pem
WARNING: For security reasons, the original certificate,
containing the private key, will be removed,
once the private key is processed.
Certificate endpoint-certificate-interca-signed.pem not installed: There should be exactly one private key (ie, private_bytes) in the pem contents.
[sysadmin@controller-0 wcp-111(keystone_admin)]$

This is applicable to:
   * -m ssl
   * -m docker_registry
   * and probably -m OpenStack

Hopefully, this is just a semantic check error in the front end of the API processing.
For the "-m ssl" certificate, the servers are haproxy and lighttpd.
From work on Secure Internal Endpoints feature, which uses intermediateCA-signed certificates, we know haproxy supports such certificates. Hopefully lighttpd and docker registry server does as well.

Severity
--------
Provide the severity of the defect.
<Minor: System/Feature is usable with minor issue>

Steps to Reproduce
------------------

You need to create intermediateCA-signed certificates.
You can do this with ssl or cert-manager on any system.
See me/gwaines for details on how to do it with cert-manager.

I created these intermediateCA signed certificates with cert-manager.
Notice that cert-manager correctly puts the intermediateCA certificate as well as the server certificate in the tls.crt field / file.

[sysadmin@controller-0 wcp-111(keystone_admin)]$ ls -l endpoint-certificate-interca-signed.*
-rw-r--r-- 1 sysadmin sys_protected 1070 Jun 11 20:32 endpoint-certificate-interca-signed.ca.crt
-rw-r--r-- 1 sysadmin sys_protected 2258 Jun 11 20:32 endpoint-certificate-interca-signed.tls.crt
-rw-r--r-- 1 sysadmin sys_protected 1679 Jun 11 20:32 endpoint-certificate-interca-signed.tls.key
[sysadmin@controller-0 wcp-111(keystone_admin)]$
[sysadmin@controller-0 wcp-111(keystone_admin)]$ cat endpoint-certificate-interca-signed.tls.crt
-----BEGIN CERTIFICATE-----
MIIDQDCCAiigAwIBAgIQWdKoE1qIF+vtEOJESDh9rTANBgkqhkiG9w0BAQsFADAg
MR4wHAYDVQQDExVjbG91ZHBsYXRmb3JtLWludGVyY2EwHhcNMjAwNjExMTA1NjUz
WhcNMjAwOTA5MTA1NjUzWjBDMRYwFAYDVQQKEw1DbG91ZFBsYXRmb3JtMSkwJwYD
VQQDEyBjZ2NzLXdpbGRjYXQtMTExLmN1bXVsdXMud3JzLmNvbTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBANwGWkXR6oLQ9ip+6n8SQ5f9C9iHARX6qhy/
ot3us3yQJETfH2lC7C+1JsKFxBn+voLjTupYoCVOpdesmAZTlcbM4roKCcBbs/ta
g8AuRufBWrWpeHrHd+kLGPoEukNHz0NaCI0ou/uiGj4LmwsX0yzZqd8RpGcTadVs
wxpzoXpw1wO4ZLI2qjuHdyBxRL1vz5LOxHoXt+niHDAwCqWPj6Ahzep0SFq7KwLv
rzlB7PEFWarF8/UdGFJSYtZB46XQeu5GveF5efnDv/W4yhtbQBmzeK9MhKBVUQ3G
esE2PtCxKFspHXVkKJhwEXCaLUl+zEP5qb5JfLLClPQnoELNGuECAwEAAaNTMFEw
DgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwMQYDVR0RBCowKIIgY2djcy13
aWxkY2F0LTExMS5jdW11bHVzLndycy5jb22HBIDglzkwDQYJKoZIhvcNAQELBQAD
ggEBAMeFJIbdNpBpo7+fmz9pJF/xafvgB0UMQN+Wnx5/H6GouxwEKeB7FrS9Bfrz
SvmyqINtrFg3MS18yIw+/5xi5BwJa7LTJ5eBU6Q/9gVoMxvFPL9ij9wohQDo1db0
6XL+80vVibROy69fVTs0PETWOlFXF+jQWatJ+RzLvPr6Ys6y9S1OrYBfPojS0LnQ
A2h+H6QQurzA5OYaL/1AjspEX1qpq/0GhVbNFvH0d8y4/zvkflJ9FzKa9aGPoj3t
H6S8F1R2Ls4mX40/vPWC2FlllJNZFN+xSHfUPyf3CpIuwcRvzpVdzv/T0AUrRk5X
2HgpYfOMc8rB9Mk493DvZDDDdlw=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6TCCAdGgAwIBAgIRAJ7zM//CYx1urUNK6mGg3gQwDQYJKoZIhvcNAQELBQAw
GzEZMBcGA1UEAxMQd2luZHJpdmVyLXJvb3RjYTAeFw0yMDA2MTExMDU2MzRaFw0y
MDA5MDkxMDU2MzRaMCAxHjAcBgNVBAMTFWNsb3VkcGxhdGZvcm0taW50ZXJjYTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM9qsgrqcSo777oowDB43TPh
CLPLOgnGxh7dqsxVQHIMGk1QrzrNRi8CmyYs6I0po4QsbVwTdDXcnCkaXXX8E7Hl
dUrzsp39Lb4o5PY0LMRo9I76AJDfJT66RMO8oJ22JaPPD20DooRAldNWc86klEre
U0v9izGBZaTDy8rr4wKvtnjvXNv+tTv92dFWqIwIu0p/XnPpdfnsX2cQZkMRFuwj
jfXPiv5fkgbyQ4lyivNfVRNF2FKsIrOTnTko6rFk/2xAlNvaqHLK6HuJq0F8JnI7
VEtFvqbSSQFE6/jI8s/62YkbYUNrJ7Uq8kwvlsgt7yuOyuWQaAj1i59PJV1NsR0C
AwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAID2FhbAxTbExMTVEvoZVbGY6LdYh8kbFGfP/uOLolkpfye3
KQYm0v6A6ccZWa3/ejd6eJzAlDw3zcVftWYy4f8mUnK86+20a4gxT6T0P0/pKkjY
wdmUb7poKX5ey5nRncfkODrAm/T26vMX7x6Jt8CbaefXHaCdZj5JF8Vatc4NVz7+
pRQWPptysHzZ4XrdBE6CrUp3+MHeKDyexhpwh1xVoK79555eGuTe/yvIrATPG2rn
l/LZlrwlPdKWxQMC76Gm6m4NQ4cS11/ftUai2TQHmugqFJobNLNFCtsdyN79VR5w
fW1LGc7rzAb82XMNYzMFPwE+JkhijmIgjmA4g0c=
-----END CERTIFICATE-----
[sysadmin@controller-0 wcp-111(keystone_admin)]$

In order to call ‘system certificate-install’ you have to concatenate the .crt file and the .key file.

[sysadmin@controller-0 wcp-111(keystone_admin)]$ cat endpoint-certificate-interca-signed.tls.crt endpoint-certificate-interca-signed.tls.key > endpoint-certificate-interca-signed.pem
[sysadmin@controller-0 wcp-111(keystone_admin)]$
[sysadmin@controller-0 wcp-111(keystone_admin)]$ cat endpoint-certificate-interca-signed.pem
-----BEGIN CERTIFICATE-----
MIIDQDCCAiigAwIBAgIQWdKoE1qIF+vtEOJESDh9rTANBgkqhkiG9w0BAQsFADAg
MR4wHAYDVQQDExVjbG91ZHBsYXRmb3JtLWludGVyY2EwHhcNMjAwNjExMTA1NjUz
WhcNMjAwOTA5MTA1NjUzWjBDMRYwFAYDVQQKEw1DbG91ZFBsYXRmb3JtMSkwJwYD
VQQDEyBjZ2NzLXdpbGRjYXQtMTExLmN1bXVsdXMud3JzLmNvbTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBANwGWkXR6oLQ9ip+6n8SQ5f9C9iHARX6qhy/
ot3us3yQJETfH2lC7C+1JsKFxBn+voLjTupYoCVOpdesmAZTlcbM4roKCcBbs/ta
g8AuRufBWrWpeHrHd+kLGPoEukNHz0NaCI0ou/uiGj4LmwsX0yzZqd8RpGcTadVs
wxpzoXpw1wO4ZLI2qjuHdyBxRL1vz5LOxHoXt+niHDAwCqWPj6Ahzep0SFq7KwLv
rzlB7PEFWarF8/UdGFJSYtZB46XQeu5GveF5efnDv/W4yhtbQBmzeK9MhKBVUQ3G
esE2PtCxKFspHXVkKJhwEXCaLUl+zEP5qb5JfLLClPQnoELNGuECAwEAAaNTMFEw
DgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwMQYDVR0RBCowKIIgY2djcy13
aWxkY2F0LTExMS5jdW11bHVzLndycy5jb22HBIDglzkwDQYJKoZIhvcNAQELBQAD
ggEBAMeFJIbdNpBpo7+fmz9pJF/xafvgB0UMQN+Wnx5/H6GouxwEKeB7FrS9Bfrz
SvmyqINtrFg3MS18yIw+/5xi5BwJa7LTJ5eBU6Q/9gVoMxvFPL9ij9wohQDo1db0
6XL+80vVibROy69fVTs0PETWOlFXF+jQWatJ+RzLvPr6Ys6y9S1OrYBfPojS0LnQ
A2h+H6QQurzA5OYaL/1AjspEX1qpq/0GhVbNFvH0d8y4/zvkflJ9FzKa9aGPoj3t
H6S8F1R2Ls4mX40/vPWC2FlllJNZFN+xSHfUPyf3CpIuwcRvzpVdzv/T0AUrRk5X
2HgpYfOMc8rB9Mk493DvZDDDdlw=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6TCCAdGgAwIBAgIRAJ7zM//CYx1urUNK6mGg3gQwDQYJKoZIhvcNAQELBQAw
GzEZMBcGA1UEAxMQd2luZHJpdmVyLXJvb3RjYTAeFw0yMDA2MTExMDU2MzRaFw0y
MDA5MDkxMDU2MzRaMCAxHjAcBgNVBAMTFWNsb3VkcGxhdGZvcm0taW50ZXJjYTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM9qsgrqcSo777oowDB43TPh
CLPLOgnGxh7dqsxVQHIMGk1QrzrNRi8CmyYs6I0po4QsbVwTdDXcnCkaXXX8E7Hl
dUrzsp39Lb4o5PY0LMRo9I76AJDfJT66RMO8oJ22JaPPD20DooRAldNWc86klEre
U0v9izGBZaTDy8rr4wKvtnjvXNv+tTv92dFWqIwIu0p/XnPpdfnsX2cQZkMRFuwj
jfXPiv5fkgbyQ4lyivNfVRNF2FKsIrOTnTko6rFk/2xAlNvaqHLK6HuJq0F8JnI7
VEtFvqbSSQFE6/jI8s/62YkbYUNrJ7Uq8kwvlsgt7yuOyuWQaAj1i59PJV1NsR0C
AwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAID2FhbAxTbExMTVEvoZVbGY6LdYh8kbFGfP/uOLolkpfye3
KQYm0v6A6ccZWa3/ejd6eJzAlDw3zcVftWYy4f8mUnK86+20a4gxT6T0P0/pKkjY
wdmUb7poKX5ey5nRncfkODrAm/T26vMX7x6Jt8CbaefXHaCdZj5JF8Vatc4NVz7+
pRQWPptysHzZ4XrdBE6CrUp3+MHeKDyexhpwh1xVoK79555eGuTe/yvIrATPG2rn
l/LZlrwlPdKWxQMC76Gm6m4NQ4cS11/ftUai2TQHmugqFJobNLNFCtsdyN79VR5w
fW1LGc7rzAb82XMNYzMFPwE+JkhijmIgjmA4g0c=
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA3AZaRdHqgtD2Kn7qfxJDl/0L2IcBFfqqHL+i3e6zfJAkRN8f
aULsL7UmwoXEGf6+guNO6ligJU6l16yYBlOVxsziugoJwFuz+1qDwC5G58Fatal4
esd36QsY+gS6Q0fPQ1oIjSi7+6IaPgubCxfTLNmp3xGkZxNp1WzDGnOhenDXA7hk
sjaqO4d3IHFEvW/Pks7Eehe36eIcMDAKpY+PoCHN6nRIWrsrAu+vOUHs8QVZqsXz
9R0YUlJi1kHjpdB67ka94Xl5+cO/9bjKG1tAGbN4r0yEoFVRDcZ6wTY+0LEoWykd
dWQomHARcJotSX7MQ/mpvkl8ssKU9CegQs0a4QIDAQABAoIBAQCk2EmYF+74dbCJ
JC9yZu+nltniLzu8COKcyWMvnaByCdhylSyGg7JLCBDzGp9Szo5TqBQ+9Yl+ig8s
qE36J3OGz4hq1Ax8RvhwsF8XhFRxXGcmxcLTUlDflAXxg8s9jzaVAoZ3o/Y7UDcT
F0ThKcd4yddAnoorA9t+II+IaOupvC9S3pYnOGn7rQ40Erj9uI5j06/na/ssmhFs
EBp3bx3JvrjhF9Aj06yNE9wFtfcm0LDdtvLmafTBo8U3Xcl7ltPtHxch7bV2jntl
PzVRlexw0kpbY7WwJPUqJ7rhmIpxlxuOVXUVmElL715xraDEV96k3r3fxjazgTMe
kEFvgno1AoGBAPLaYYsorppxmiImR5c9h1h3MEcEOosbS/pAZTUmL5sRYa50izEP
uBKGXr53a3/STHhdSZyscIUqadFoICulm+To+95LXu0zH9CNbcLUy+z67doIVnPT
TU8AwiktMcn08xawULBq2ZWHs97VVwpbPENi8DZpuQKmlgHQOrWFzz8HAoGBAOfv
mkzBujuCwQq07g2TgPDLbmu3rCRSOMZwpbGXyamsYKVEXayd0qx0M4EjbbLRh0CL
k79ZKnsKCplUDB5267SkKQzuH3mP2XRaophyurvy9QGrBbjbMukI41zGjqbY2JCC
nbjj89SAfP6lZbuG1OEaJsYgiTTQ05wCmOXBUHTXAoGAO83xlX5oORpetagtmx4I
1Y1Mc0CkVhtbw33WLpd/w7UtE/QZTvSrBMzYShlIfMrCECYQ+acxcmm2c14sBfOb
h2ykkeKHtuwPGqGpdQBIU8ybhETnXVCcyibKqa86gpxyvJSg2kEF4h/v+KEa9S7w
kTutIgk5CWESsdGgk5EnL0MCgYEAjpC9K8k5DDOHLWmhmj6810Z0jUnNSlYwB+pe
Lxe6Ic2PaobnfZ9VTCPxNMPuPiiFFqu1Gb1ZW2AEu9Lp0CARKmxtv0Dvn+1IITlF
EKgVGEpzR580WTiDxrJr7y8tqGlKVSm0gV29FzYTnwNT0T1DxUop40dEj259ZDxE
d5WoOukCgYAbszcAq8aU99TRIZCQ5YgdP9uvpEbIIuSn7EOkMCB++SU58VUYYrwV
qLrWxXR95o7EXW5CdmiQrNfCjCvZ0eh1H2Zwiqobe/WaRpoP1prGmj794OhwBi+y
GayMnK7nr1n9iMBMPMHRFb20shxcTDXp47JhV5rJC1KR9o99blELkA==
-----END RSA PRIVATE KEY-----
[sysadmin@controller-0 wcp-111(keystone_admin)]$

But “system certificate-install” fails due to incorrect semantic checks.
It actually thinks there are two keys ... when there are really two certs.

[sysadmin@controller-0 wcp-111(keystone_admin)]$ system certificate-install -m ssl endpoint-certificate-interca-signed.pem
WARNING: For security reasons, the original certificate,
containing the private key, will be removed,
once the private key is processed.
Certificate endpoint-certificate-interca-signed.pem not installed: There should be exactly one private key (ie, private_bytes) in the pem contents.
[sysadmin@controller-0 wcp-111(keystone_admin)]$
[sysadmin@controller-0 wcp-111(keystone_admin)]$

The code needs to separate the certs into one file and the private key into the other file.
It is valid to check that there is only one private key ... but there can be multiple certs.

Expected Behavior
------------------
Should accept the above pem file with 2x certs and 1x key.
And when client connects to the https server, the server should return both these certs to the client for certificate validation. And then client, with just the rootCA cert as trusted, should be able to validate the intermediateCA-signed certificate.

Actual Behavior
----------------
system certificate-install fails because it doesn't like the 2x certs in the pem file.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
All configs.

Branch/Pull Time/Commit
-----------------------
Current early june loads.

Last Pass
---------
Never

Timestamp/Logs
--------------
NA

Test Activity
-------------
Evaluation

 Workaround
 ----------
Use only rootCA-signed certificates.

See original description
Ghada Khalil (gkhalil)
tags: added: stx.security
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
tags: added: stx.5.0
Revision history for this message
Ghada Khalil (gkhalil) wrote :

This is an improvement on what starlingx supports today, so we'll consider it for stx.5.0

Changed in starlingx:
assignee: nobody → Ghada Khalil (gkhalil)
Ghada Khalil (gkhalil)
Changed in starlingx:
assignee: Ghada Khalil (gkhalil) → Andy (andy.wrs)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/740958

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/740960

Ghada Khalil (gkhalil)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/740958
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=51cd539e5d3ecd21bb94756d0a56b8b5bff25320
Submitter: Zuul
Branch: master

commit 51cd539e5d3ecd21bb94756d0a56b8b5bff25320
Author: Andy Ning <email address hidden>
Date: Mon Jun 29 17:08:27 2020 -0400

    Update lighttpd config to support ICA signed certificate

    Updated lighttpd puppet template to generate configuration file
    that supports intermediate CA signed certificate.

    Change-Id: Id7d10d91a7ba7c35bcaf3056824e80410e72cfb8
    Closes-Bug: 1883300
    Signed-off-by: Andy Ning <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
status: Fix Released → In Progress
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Re-opening as the second commit hasn't merged yet

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/740960
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=87469ebfa99ae75651359941770ceede74382a21
Submitter: Zuul
Branch: master

commit 87469ebfa99ae75651359941770ceede74382a21
Author: Andy Ning <email address hidden>
Date: Mon Jun 29 17:02:07 2020 -0400

    Support ICA signed certificate installation

    Enhanced certificate installation to support intermediate CA signed
    ssl, docker_registry, openstack certificate.

    Change-Id: I6ed9dfbf3f3379ad152d714dac691ea16f7f206e
    Depends-On: https://review.opendev.org/#/c/740958/
    Closes-Bug: 1883300
    Signed-off-by: Andy Ning <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to docs (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/745168

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to docs (master)

Reviewed: https://review.opendev.org/745168
Committed: https://git.openstack.org/cgit/starlingx/docs/commit/?id=fccb07668aaaefaeeae5991ecb9322b1b1ab3edd
Submitter: Zuul
Branch: master

commit fccb07668aaaefaeeae5991ecb9322b1b1ab3edd
Author: Andy Ning <email address hidden>
Date: Thu Aug 6 11:03:05 2020 -0400

    Update certificate config for ICA signed certificate

    Updated certificate installation in cert config to include ICA signed
    certificate.

    Change-Id: Ia7ba930ad1211ef72b4544d672a56ab5bbb844c8
    Related-Bug: 1883300
    Signed-off-by: Andy Ning <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to docs (r/stx.4.0)

Related fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/747779

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to docs (r/stx.4.0)

Reviewed: https://review.opendev.org/747779
Committed: https://git.openstack.org/cgit/starlingx/docs/commit/?id=28750704c1cf22cf2ddc2d8be26fde25f16fd613
Submitter: Zuul
Branch: r/stx.4.0

commit 28750704c1cf22cf2ddc2d8be26fde25f16fd613
Author: Andy Ning <email address hidden>
Date: Thu Aug 6 11:03:05 2020 -0400

    Update certificate config for ICA signed certificate

    Updated certificate installation in cert config to include ICA signed
    certificate.

    Change-Id: Ia7ba930ad1211ef72b4544d672a56ab5bbb844c8
    Related-Bug: 1883300
    Signed-off-by: Andy Ning <email address hidden>
    (cherry picked from commit fccb07668aaaefaeeae5991ecb9322b1b1ab3edd)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/762919

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.