IPv6 Distributed Cloud: new created certificate not sync to subcloud

Bug #1851252 reported by Peng Peng
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Andy

Bug Description

Brief Description
-----------------
In DC system, created a new certificate on System controller, but the new certificate does not sync to subcloud.

Severity
--------
Major

Steps to Reproduce
------------------
On System Controller
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 1024 -out ca-cert.pem -outform PEM
openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -out server.csr
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out server.pem -days 365
vi extfile.cnf
openssl x509 -req -days 365 -in server.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out server.pem -extfile extfile.cnf
cat server-key.pem server.pem > server-with-key.pem
system certificate-install -m docker_registry server-with-key.pem

TC-name:

Expected Behavior
------------------
new certificate sync to all in-sync subcloud

Actual Behavior
----------------

Reproducibility
---------------
Seen once

System Configuration
--------------------
Distributed Cloud system

Lab-name:

Branch/Pull Time/Commit
-----------------------
2019-11-02_08-39-54

Last Pass
---------

Timestamp/Logs
--------------
System Controller:
 updated_at | 2019-11-03T02:56:47.674281 |
+-----------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud list
+----+-----------+------------+--------------+---------------+---------+
| id | name | management | availability | deploy status | sync |
+----+-----------+------------+--------------+---------------+---------+
| 3 | subcloud6 | managed | online | complete | in-sync |
| 4 | subcloud5 | managed | online | complete | in-sync |
| 5 | subcloud1 | managed | online | complete | in-sync |
| 6 | subcloud4 | managed | online | complete | in-sync |
+----+-----------+------------+--------------+---------------+---------+
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list

[sysadmin@controller-0 ~(keystone_admin)]$ system certificate-install -m docker_registry server-with-key.pem
WARNING: For security reasons, the original certificate,
containing the private key, will be removed,
once the private key is processed.
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | e8f82bd5-3271-4a8b-8a92-fd6c12ad1ab9 |
| certtype | docker_registry |
| signature | docker_registry_10786663268586284892 |
| start_date | 2019-11-03 03:13:57+00:00 |
| expiry_date | 2020-11-02 03:13:57+00:00 |
+-------------+--------------------------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system certificate-list
+--------------------------------------+------------+---------------------------+
| uuid | certtype | expiry_date |
+--------------------------------------+------------+---------------------------+
| 63d88610-3671-4bdd-9a72-40f66b73e4a3 | ssl_ca | 2021-06-05T20:28:20+00:00 |
| e8f82bd5-3271-4a8b-8a92-fd6c12ad1ab9 | docker_reg | 2020-11-02T03:13:57+00:00 |
| | istry | |
| | | |
+--------------------------------------+------------+---------------------------+

Subcloud6:
[sysadmin@controller-0 ~(keystone_admin)]$ system certificate-list
+--------------------------------------+----------+---------------------------+
| uuid | certtype | expiry_date |
+--------------------------------------+----------+---------------------------+
| 51cf6ee8-f161-4bd2-805b-247460bd5871 | ssl_ca | 2021-06-05T20:28:20+00:00 |
+--------------------------------------+----------+---------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ date
Mon Nov 4 15:03:27 UTC 2019

Test Activity
-------------
Sanity

Revision history for this message
Peng Peng (ppeng) wrote :
Revision history for this message
Peng Peng (ppeng) wrote :
Yang Liu (yliu12)
tags: added: stx.retestneeded
Revision history for this message
Ghada Khalil (gkhalil) wrote :

@Andy, Are these certificates supposed to be sync'd to the subclouds?

tags: added: stx.distcloud
Revision history for this message
Peng Peng (ppeng) wrote :

Installed ca-cert.pem on subcloud, but pull image still failed

subcloud1:
[sysadmin@controller-1 ~(keystone_admin)]$ system certificate-install -m ssl_ca ca-cert.pem
WARNING: For security reasons, the original certificate,
containing the private key, will be removed,
once the private key is processed.
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | 8b0b2a83-dec1-4dc2-ba50-42d32a073221 |
| certtype | ssl_ca |
| signature | ssl_ca_9661015381606263431 |
| start_date | 2019-11-03 03:09:37+00:00 |
| expiry_date | 2022-08-23 03:09:37+00:00 |
+-------------+--------------------------------------+
[sysadmin@controller-1 ~(keystone_admin)]$
[sysadmin@controller-1 ~(keystone_admin)]$
[sysadmin@controller-1 ~(keystone_admin)]$ system certificate-list
+--------------------------------------+----------+---------------------------+
| uuid | certtype | expiry_date |
+--------------------------------------+----------+---------------------------+
| 8b0b2a83-dec1-4dc2-ba50-42d32a073221 | ssl_ca | 2022-08-23T03:09:37+00:00 |
+--------------------------------------+----------+---------------------------+
[sysadmin@controller-1 ~(keystone_admin)]$ system certificate-show 8b0b2a83-dec1-4dc2-ba50-42d32a073221
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | 8b0b2a83-dec1-4dc2-ba50-42d32a073221 |
| certtype | ssl_ca |
| signature | ssl_ca_9661015381606263431 |
| start_date | 2019-11-03T03:09:37+00:00 |
| expiry_date | 2022-08-23T03:09:37+00:00 |
+-------------+--------------------------------------+
[sysadmin@controller-1 ~(keystone_admin)]$
[sysadmin@controller-1 ~(keystone_admin)]$ cat wind-river-cloud-platform-deployment-manager-overrides-subcloud.yaml
manager:
  image:
    repository: registry.central:9001/tis-lab-registry.cumulus.wrs.com/wind-river/cloud-platform-deployment-manager
    tag: TC_19.10
    pullPolicy: IfNotPresent
[sysadmin@controller-1 ~(keystone_admin)]$ sudo docker pull registry.central:9001/tis-lab-registry.cumulus.wrs.com/wind-river/cloud-platform-deployment-manager
Password:
Using default tag: latest
Error response from daemon: Get https://registry.central:9001/v2/tis-lab-registry.cumulus.wrs.com/wind-river/cloud-platform-deployment-manager/manifests/latest: Get https://[2620:10a:a001:a103::1162]:9002/token/?scope=repository%3Atis-lab-registry.cumulus.wrs.com%2Fwind-river%2Fcloud-platform-deployment-manager%3Apull&service=%5Bfd01%3A1%3A%3A2%5D%3A9001: x509: certificate is valid for 192.168.204.2, 2620:10a:a001:a103::1151, not 2620:10a:a001:a103::1162

Revision history for this message
Andy (andy.wrs) wrote :

No, synchronization of docker_registry certificate is not supported. Only ssl and ssl_ca certificate synchronizations are supported.

However even ssl and ssl_ca synchronization is broken now. It's broken by the removal of tpmconfig from sysinv API by:
https://opendev.org/starlingx/config/commit/e775b01bc1fc2d778a1cca8d9cd99a67b7847a77

And by the cleanup of Murano by:
https://opendev.org/starlingx/config/commit/a974b35ce404774649d6dbc73943c7ce2fe4b6ce

Revision history for this message
Ghada Khalil (gkhalil) wrote :

This bug will be used to address the sync issue or the ssl & ssl_ca certificates and to add sync capability for the docker-registry certificate.

stx.3.0 / high priority - related for distributed cloud feature which is feature deliverable for this release

tags: added: stx.3.0
Changed in starlingx:
importance: Undecided → High
status: New → Triaged
assignee: nobody → Andy (andy.wrs)
Andy (andy.wrs)
Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/695308

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to distcloud (master)

Fix proposed to branch: master
Review: https://review.opendev.org/695309

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/695308
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=6adf744195e544caf4f018885bca1a1a4d0958ce
Submitter: Zuul
Branch: master

commit 6adf744195e544caf4f018885bca1a1a4d0958ce
Author: Andy Ning <email address hidden>
Date: Wed Nov 20 15:56:42 2019 -0500

    Change subcloud local registry token auth realm to registry.local

    In DC system, subcloud local registry token realm is currently set to
    oam floating IP address. When a new docker registry certificate is
    installed in central cloud and get synced to subcloud, access to local
    registry in subcloud is broken during certificate verfication, with error
    like the following:

    Error response from daemon: Get https://registry.local:9001/v2/: Get
    https://10.10.10.13:9002/token/?account=admin&client_id=docker&
    offline_token=true&service=192.168.101.2%3A9001:
    x509: certificate is valid for 192.168.204.2, 10.10.10.3, not
    10.10.10.13

    This change updated subcloud local registry's token auth realm to be
    domain name registry.local so the certificate verfication is against
    the DNS names in the certifcate. This makes the verfication succeed.

    Change-Id: I710e2e27461276db90cd9f9275655b5ecf00e342
    Closes-Bug: 1851252
    Signed-off-by: Andy Ning <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Re-opening until the second commit merges

Changed in starlingx:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to distcloud (master)

Reviewed: https://review.opendev.org/695309
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=2f5eb38cb2e7de65e13f2b487b29e8c89ca6531e
Submitter: Zuul
Branch: master

commit 2f5eb38cb2e7de65e13f2b487b29e8c89ca6531e
Author: Andy Ning <email address hidden>
Date: Wed Nov 20 16:23:40 2019 -0500

    DC sync ssl and docker registry certificates

    This change added support to synchronize ssl and docker registry
    certificates from central cloud to subclouds.

    Change-Id: I4cdcf32264d8e177fee3549ce17d172f9fc36c36
    Closes-Bug: 1851252
    Depends-On: https://review.opendev.org/#/c/695308
    Signed-off-by: Andy Ning <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Peng Peng (ppeng) wrote :

Verified on 2019-12-01_20-00-00

Peng Peng (ppeng)
tags: removed: stx.retestneeded
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.