Cannot install multiple trusted CA during bootstrap

Bug #1860995 reported by Jerry Sun
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Andy

Bug Description

Brief Description
-----------------
providing multiple CA certs in the file specified by "ssl_ca_cert" during ansible bootstraps installs only the first CA cert

Severity
--------
Provide the severity of the defect.
Minor

Steps to Reproduce
------------------
Generate 2 CA and concat their certificates together. Specify this file during bootstrap under "ssl_ca_cert".
Run bootstrap.
Have each CA sign something (a public key for a docker registry for example, or create a dummy key for the CAs to sign)
use "openssl verify <cert you created in previous step>" to try and verify the certificate.

Expected Behavior
------------------
openssl verify should say the certificate is OK

Actual Behavior
----------------
When verifying the certificate signed by the second CA specified by "ssl_ca_cert" during bootstrap, verification fails with "unable to get local issuer certificate".
The system does not recognize the second CA specified during bootstrap under "ssl_ca_cert" as a trusted CA
we know this because if you specify the CA file when running openssl verify (openssl verify -CAfile ca-cert.pem server.pem, where ca-cert.pem is the CA's certificate and server.pem is the dummy file that the CA signed), the verification is successful.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
Master as of Jan 27

Last Pass
---------

Timestamp/Logs
--------------

Test Activity
-------------
Developer Testing

Workaround
----------
copy ca-cert.pem to /etc/pki/ca-trust/source/anchors/
run update-ca-trust
restart whatever processes need the CA to be trusted
This needs to be done to different nodes depending on where your process that needs the CA to be trusted runs on, potentially controllers, potentially controllers and computes
Also, this workaround could be undone when nodes reboot, I did not test that.

Ghada Khalil (gkhalil)
tags: added: stx.security
Revision history for this message
Ghada Khalil (gkhalil) wrote :

stx.4.0 / medium priority as there is a workaround for this issue

Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
tags: added: stx.4.0
Changed in starlingx:
assignee: nobody → David Sullivan (dsullivanwr)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Assigning to Andy Ning since this seems similar to https://bugs.launchpad.net/starlingx/+bug/1861438

Changed in starlingx:
assignee: David Sullivan (dsullivanwr) → Andy (andy.wrs)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/708502

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/711538

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on config (master)

Change abandoned by Andy Ning (<email address hidden>) on branch: master
Review: https://review.opendev.org/708502
Reason: Replaced by:
https://review.opendev.org/#/c/711538/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (master)

Fix proposed to branch: master
Review: https://review.opendev.org/711633

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to distcloud (master)

Fix proposed to branch: master
Review: https://review.opendev.org/712152

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/711633
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=c03dc5f3fc7dd1a5646cfc5b83d6c6fc08241bc1
Submitter: Zuul
Branch: master

commit c03dc5f3fc7dd1a5646cfc5b83d6c6fc08241bc1
Author: Andy Ning <email address hidden>
Date: Thu Mar 5 16:57:29 2020 -0500

    Create ssl_ca dir to support multiple CA certificate

    This update creates ssl_ca directory in the drbd shared fs during
    ansible bootstrap. The directory is used to store the CA certificates
    as individual files. CA certificate install and uninstall operations
    are manipulating these certificate files.

    Change-Id: Idfca7220320a5de1cc274e077fc7ce8d2778a76b
    Closes-Bug: 1861438
    Closes-Bug: 1860995
    Signed-off-by: Andy Ning <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/711538
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=c1c18871d72cdcd877b95f593bd119b47b3ddbb6
Submitter: Zuul
Branch: master

commit c1c18871d72cdcd877b95f593bd119b47b3ddbb6
Author: Andy Ning <email address hidden>
Date: Tue Feb 18 14:52:06 2020 -0500

    Support multiple CA certificates installation

    This update enhanced sysinv certificate install API to be able to
    install multiple CA certs from a file. The returns from the API call
    indicates the certs actually installed in the call (ie, excluding these
    that are already in the system). This is neccessary especially for DC to
    support multiple CA certs synchronization.

    This update also added sysinv certficate uninstall API. The API is to
    be used to remove a particular CA certficate from the system, identified
    by its uuid. The API returns a json body with information about the
    certificate that has been removed. This is required by DC sysinv api
    proxy for certificate deletion synchronization, since DC tracks subcloud
    certificates resource by signature while the uninstall API request
    contains only uuid.

    The uninstall API only supports ssl_ca certificate.

    cgtsclient and system CLI are also updated to align with the updated
    and new APIs. User can use "system certificate-install ..." to install
    one or multiple CA certificates, and "system certificate-uninstall ..."
    to remove a particular CA certificate from the system.

    When multiple CA certificates are installed in the system,
    "system certificate-list" will display each of the individual
    certificates.

    THe sysinv certificate configuration API reference is updated with the
    new uninstall API. Unit tests are added for CA certificate install and
    delete APIs.

    Change-Id: I7dba11e56792b7d198403c436c37f71d7b7193c9
    Depends-On: https://review.opendev.org/#/c/711633/
    Closes-Bug: 1861438
    Closes-Bug: 1860995
    Signed-off-by: Andy Ning <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to distcloud (master)

Reviewed: https://review.opendev.org/712152
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=1e588cbefa789bf8546da708470e271f01dcd593
Submitter: Zuul
Branch: master

commit 1e588cbefa789bf8546da708470e271f01dcd593
Author: Andy Ning <email address hidden>
Date: Wed Feb 26 16:04:33 2020 -0500

    Support multiple CA certificates synchronization in DC

    This update enhanced dcorch and sysinv API proxy to support multiple CA
    certificates synchronization in DC system. The support utilizes the
    updated sysinv certificate install API and the new certificate
    uninstall API.

    Closes-Bug: 1861438
    Closes-Bug: 1860995
    Depends-On: https://review.opendev.org/#/c/711538/
    Change-Id: I407314b913ae5a56bb714b39484aea3263a41d19
    Signed-off-by: Andy Ning <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716133

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716137

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to distcloud (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716140

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (f/centos8)
Download full text (12.5 KiB)

Reviewed: https://review.opendev.org/716133
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=ddcb11f4b773f4b3190663defe3ba0f3ec4201c8
Submitter: Zuul
Branch: f/centos8

commit bf103f3c54eb45c26d52a43c35339d1d863a42de
Author: Mihnea Saracin <email address hidden>
Date: Fri Mar 27 18:19:02 2020 +0200

    Fix B&R when the controller needs to be unlocked

    After running the restore playbook, all the applications
    should be in an uploaded state. But they are in an
    applied state instead, making the controller-0
    unable to unlock.

    Closes-Bug: 1869403
    Change-Id: I8bd9c51e250969cc334d52b78c616f9ad082afd8
    Signed-off-by: Mihnea Saracin <email address hidden>

commit 6e875971afeaf1378c2c8aeb845359459838ce30
Author: Stefan Dinescu <email address hidden>
Date: Sat Mar 21 16:57:57 2020 +0200

    Fix Netapp port conflict

    By default, the Trident Netapp service opens port 8443 for
    HTTPS REST api usage. This conflicts with the port the
    Horizon dashboard uses on an HTTPS enabled setup (the port
    is also 8443).

    In order to fix this, we change the default port from 8443
    to 8678, but also make it configurable through ansible
    overrides.

    The Trident service also opens port 8001 for metrics usage.
    While that doesn't currently conflict with any other service
    on the system, I also made that configurable through
    ansible overrides, in case such a conflict appears in the
    future.

    Change-Id: I08db939acac6082f82b9e12e932d8289c7cecdeb
    Closes-bug: 1868382
    Signed-off-by: Stefan Dinescu <email address hidden>

commit 5a9ba6786e393f2cd93bfae8c3a8f09f0cf9eb26
Author: Robert Church <email address hidden>
Date: Thu Mar 19 19:08:17 2020 -0400

    Upversion Multus to 3.4

    Updates the Multus configuration to align with version 3.4

    Change-Id: Ifc236ccbbe4e559987d7ef522902f638062348ca
    Depends-On: https://review.opendev.org/#/c/714024/
    Story: 2006999
    Task: 39110
    Signed-off-by: Robert Church <email address hidden>

commit 6a261463f9ac0f81d9c7f054dd3cb10a51934d4a
Author: Robert Church <email address hidden>
Date: Wed Mar 18 22:01:03 2020 -0400

    Upversion Calico from 3.6 to 3.12

    Updates the Calico configuration to align with version 3.12. This
    introduces support for a Flex Volume Driver which requires enabling the
    --volume-plugin-dir option for kubelet, the --flex-volume-plugin-dir
    option for kube-controller-manager, and pulling the pod2daemon-flexvol
    image used by calico-node pods.

    Change-Id: I74bc5c53ffcb16c8e3c06cebf20eac296b9ccc65
    Story: 2006999
    Task: 39109
    Depends-On: https://review.opendev.org/#/c/714023
    Signed-off-by: Robert Church <email address hidden>

commit b35387f8bc40714e9633e6191267284b8af8ccee
Author: Stefan Dinescu <email address hidden>
Date: Thu Mar 19 18:13:26 2020 +0200

    Netapp: Fix handling of IPv6 addresses

    Using bash process subtitution to pass the file parameter
    to the "create backend" command doesn't work as the bash
    variable expansion...

tags: added: in-f-centos8
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (f/centos8)
Download full text (32.3 KiB)

Reviewed: https://review.opendev.org/716137
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=cb4cf4299c2ec10fb2eb03cdee3f6d78a6413089
Submitter: Zuul
Branch: f/centos8

commit 16477935845e1c27b4c9d31743e359b0aa94a948
Author: Steven Webster <email address hidden>
Date: Sat Mar 28 17:19:30 2020 -0400

    Fix SR-IOV runtime manifest apply

    When an SR-IOV interface is configured, the platform's
    network runtime manifest is applied in order to apply the virtual
    function (VF) config and restart the interface. This results in
    sysinv being able to determine and populate the puppet hieradata
    with the virtual function PCI addresses.

    A side effect of the network manifest apply is that potentially
    all platform interfaces may be brought down/up if it is determined
    that their configuration has changed. This will likely be the case
    for a system which configures SR-IOV interfaces before initial
    unlock.

    A few issues have been encountered because of this, with some
    services not behaving well when the interface they are communicating
    over suddenly goes down.

    This commit makes the SR-IOV VF configuration much more targeted
    so that only the operation of setting the desired number of VFs
    is performed.

    Closes-Bug: #1868584
    Depends-On: https://review.opendev.org/715669
    Change-Id: Ie162380d3732eb1b6e9c553362fe68cbc313ae2b
    Signed-off-by: Steven Webster <email address hidden>

commit 45c9fe2d3571574b9e0503af108fe7c1567007db
Author: Zhipeng Liu <email address hidden>
Date: Thu Mar 26 01:58:34 2020 +0800

    Add ipv6 support for novncproxy_base_url.

    For ipv6 address, we need url with below format
    [ip]:port

    Partial-Bug: 1859641

    Change-Id: I01a5cd92deb9e88c2d31bd1e16e5bce1e849fcc7
    Signed-off-by: Zhipeng Liu <email address hidden>

commit d119336b3a3b24d924e000277a37ab0b5f93aae1
Author: Andy Ning <email address hidden>
Date: Mon Mar 23 16:26:21 2020 -0400

    Fix timeout waiting for CA cert install during ansible replay

    During ansible bootstrap replay, the ssl_ca_complete_flag file is
    removed. It expects puppet platform::config::runtime manifest apply
    during system CA certificate install to re-generate it. So this commit
    updated conductor manager to run that puppet manifest even if the CA cert
    has already installed so that the ssl_ca_complete_flag file is created
    and makes ansible replay to continue.

    Change-Id: Ic9051fba9afe5d5a189e2be8c8c2960bdb0d20a4
    Closes-Bug: 1868585
    Signed-off-by: Andy Ning <email address hidden>

commit 24a533d800b2c57b84f1086593fe5f04f95fe906
Author: Zhipeng Liu <email address hidden>
Date: Fri Mar 20 23:10:31 2020 +0800

    Fix rabbitmq could not bind port to ipv6 address issue

    When we use Armada to deploy openstack service for ipv6, rabbitmq
    pod could not start listen on [::]:5672 and [::]:15672.
    For ipv6, we need an override for configuration file.

    Upstream patch link is:
    https://review.opendev.org/#/c/714027/

    Test pass for deploying rabbitmq service on both ipv...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to distcloud (f/centos8)
Download full text (15.6 KiB)

Reviewed: https://review.opendev.org/716140
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=04b49dd093ab850f4520cdb85638221120dd7568
Submitter: Zuul
Branch: f/centos8

commit 25c9d6ed3861f2d783404fcf84b186441ab9cd4d
Author: albailey <email address hidden>
Date: Wed Mar 25 15:43:32 2020 -0500

    Removing ddt from unit tests

    This cleanup should assist in transitioning to
    stestr and fixtures, as well as py3 support.

    The ddt data is primarily unused, only subcloud, route
    and endpoints were being loaded.

    The information in the data files was out of date,
    and not necessarily matching the current product model.

    Story: 2004515
    Task: 39160
    Change-Id: Iddd7ed4664b0d59dbc58aae5c3fedd74c9a138c0
    Signed-off-by: albailey <email address hidden>

commit 7f3827f24d2fb3cb546d3caf71d505d23187b0dc
Author: Tao Liu <email address hidden>
Date: Thu Mar 12 09:46:29 2020 -0400

    Keystone token and resource caching

    Add the following misc. changes to dcorch and dcmanager components:
    - Cache the master resource in dcorch audit
    - Consolidate the openstack drivers to common module, combine the
      dcmanager and dcorch sysinv client. (Note: the sdk driver that
      used by nova, neutron and cinder will be cleaned as part of
      story 2006588).
    - Update the common sdk driver:
      . in order to avoid creating new keystone client multiple times
      . to add a option for caching region clients, in addition to the
        keystone client
      . finally, to randomize the token early renewal duration
    - Change subcloud audit manager, patch audit manager,
      and sw update manager to:
      utilize the sdk driver which caches the keystone client and token

    Test cases:
    1. Manage/unmanage subclouds
    2. Platform resources sync and audit
    3. Verify the keystone token is cached until the token is
       expired
    4. Add/delete subclouds
    5. Managed subcloud goes offline/online (power off/on)
    6. Managed subcloud goes offline/online (delete/add a static route)
    7. Apply a patch to all subclouds via patch Orchestration

    Story: 2007267
    Task: 38865

    Change-Id: I75e0cf66a797a65faf75e7c64dafb07f54c2df06
    Signed-off-by: Tao Liu <email address hidden>

commit 3a1bf60caddfa2e807d4f5996ff94fea7dde5477
Author: Jessica Castelino <email address hidden>
Date: Wed Mar 11 16:23:21 2020 -0400

    Cleanup subcloud details when subcloud add fails

    Failure during add subcloud prevents subcloud from being added again
    with the same name as the subcloud details are not cleaned up
    properly. Fixes have been added for proper cleanup of dcorch database
    tables, ansible subcloud inventory files, keystone endpoints, keystone
    region, and addn_hosts_dc file when failure is encountered.

    Test cases:
    1. Add subcloud
    2. Add subcloud with "--deploy-playbook"
    3. Delete subcloud
    4. Raise explicit exception in dcorch/objects/subcloud.py
    5. Raise explicit exception in dcmanager/manager/subcloud_manager.py

    Change-Id: Iedf172c3e9c3c4bdb9b9482dc5d46f072b3ccf61
    ...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.