Distributed Cloud: subcloud cannot use central-cloud's local registry

[Greg Waines]

Brief Description
-----------------

Currently the subclouds can not use the central cloud's local registry to pull images from for a couple of reasons.
- there is no DNS entry in the subcloud for the central-cloud's local registry ... and need to use DNS name in order to more easily use either IPv4 or IPv6, and
- need to add this DNS entry to the SANs of the central-cloud's local registry

i.e. proposed solution ... to do this automatically

* On central-cloud,
     > On bootstrap of ‘systemcontroller’ role,
          - Add ‘registry.central’ into the central-cloud’s docker registry’s certificate SAN
            ( this certificate is auto-generated )

* On subclouds
     > On bootstrap of subcloud
          - Add to dnsmasq ‘registry.central à < central-cloud’s floating management ip address >’
               o i.e. do this early so we could use ‘registry.central’ in docker registries:
                      overrides of bootstrap
                      so you could pull system images from central-cloud docker registry
           - NOTE: dnsmasq may not be available during bootstrap,
                    so might have to temporarily put this in /etc/hosts

NOTE: solution needs to make sure that both controllers/masters and workers on the subcloud can resolve registry.central and pull from the central-cloud's local registry.

Severity
--------
<Major: System/Feature is usable but degraded>

Steps to Reproduce
------------------
Configure DC and a subcloud and try to pull from central-cloud's local registry using IP Address.

Expected Behavior
------------------
Subcloud nodes should be able to pull from central-cloud's local registry.

Actual Behavior
----------------
They can not.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any/All.

Branch/Pull Time/Commit
-----------------------
NA

Last Pass
---------
NA

Timestamp/Logs
--------------
NA

Test Activity
-------------
[Evaluation]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690082

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690084

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690086

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/691525

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690082
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=8849dc98294c7410717b30ebd39e0c5ab8cdbed4
Submitter: Zuul
Branch: master

commit 8849dc98294c7410717b30ebd39e0c5ab8cdbed4
Author: Tao Liu <email address hidden>
Date: Tue Oct 22 09:44:11 2019 -0400

    Allow subcloud to use central-cloud's local registry

    Add the following to bootstrap to setup the environment
    which allows the subcloud to use central-cloud's local registry
    via the OAM interface

     - Add system controller oam floating address and subnet support
     - Populate the system controller oam address pool and network
       on subcloud
     - Add 'registry.central' to the host file on subcloud until dnsmasq
       takes over after the first unlock
     - Copy the central-cloud's local registry certificate to the
       subcloud's docker certificate directory
     - Add docker registry public IP and realm host support
     - Add 'registry.central' and system_controller_oam_floating_address
       to the default_no_proxy list on subcloud

    Change-Id: I05bdbac30c25da371ffa879babfc8df2e88f2812
    Partial-Bug: 1846799
    Signed-off-by: Tao Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690084
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=0d8df17df6ae911e8978fa4da19285b09d5e9f90
Submitter: Zuul
Branch: master

commit 0d8df17df6ae911e8978fa4da19285b09d5e9f90
Author: Tao Liu <email address hidden>
Date: Tue Oct 22 09:57:50 2019 -0400

    Allow subcloud to use central-cloud's local registry

    Add the following to setup the environment which allows the subcloud
    to use central-cloud's local registry via the OAM interface

     - controller_config:
       copy registry.central certificate from the shared directory to
       docker certificate directory

     - worker_config:
       copy registry.central certificate from the shared directory to
       docker certificate directory

     - sysinv:
       Add a new network type for system controller OAM network
       Retrieve the system controller's OAM floating IP address
       from DB and populate the hiera record for dnsmasq
       Add a public URL encoded address for haproxy

    Depends-On: https://review.opendev.org/#/c/690082/
    Change-Id: Ibbc7f0ed84679a3ced3a9fee712bd1da5865f213
    Partial-Bug: 1846799
    Signed-off-by: Tao Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690086
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=f274e1391bbcf4802ac6370e48de9a021153db2e
Submitter: Zuul
Branch: master

commit f274e1391bbcf4802ac6370e48de9a021153db2e
Author: Tao Liu <email address hidden>
Date: Tue Oct 22 10:01:51 2019 -0400

    Allow subcloud to use central-cloud's local registry

    Update the dnsmasq config with the following change which allows
    the subcloud to use central-cloud's local registry via the OAM
    interface

     - Add a CNAME record on central cloud which indicates the
       'registry.central' is the 'controller'
     - Add 'registry.central' domain to return the system controller
       OAM IP address on subcloud
     - Add docker registry and token server ports to OAM Firewall
     - Add docker registry and token server entries in HAPROXY with
       passthrough configuration
     - Add 'registry.central' and the OAM IP into the docker registry
       certificate SAN
     - Configure the docker auth token realm URL using the public URL
       encoded address

    Test cases:
    Non-DC:
    1. AIO-Simplex: auto-install and provisioning
    2. AIO-Duplex: auto-install and provisioning
    3. Enable https and verify the haproxy configuration
    4. Verify docker login to registry.local
    DC:
    1. System controller installation and configuration
    2. AIO-Simplex subcloud bootstrapping without http proxy
    3. AIO-Duplex subcloud bootstrapping with http proxy and the
       docker registry override using the registry.central
    4. Standard system (controller + worker) bootstrapping with the
       docker registry override using the registry.central
    5. On subcloud controller, pull an image from the registry.central
       via the OAM interface, it also verifies the firewall change
    6. On subcloud worker node, pull an image from the registy.central
    7. Install a custom certificate during system controller bootstrap
       and verified the subcloud bootstrapping using the
       registry.central
    8. Install a new certificate in the central cloud
       system certificate-install -m docker_registry <path to cert>
       Install the customer_ca on the subclouds as a trusted ca
       system certificate-install -m ssl_ca ca-cert.pem
       Verified docker login to the registry.central

    Depends-On: https://review.opendev.org/#/c/690084/
    Change-Id: I22f88183200a5b16a62773efba520b19e3ebe725
    Closes-Bug: 1846799
    Signed-off-by: Tao Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691525
Committed: https://git.openstack.org/cgit/starlingx/distcloud/commit/?id=20a06bded7e6d0383371cacf4369cc1a8656c850
Submitter: Zuul
Branch: master

commit 20a06bded7e6d0383371cacf4369cc1a8656c850
Author: Tao Liu <email address hidden>
Date: Fri Oct 25 10:50:35 2019 -0400

    Allow subcloud to use central-cloud's local registry

    To address a requirement change that exposes the 'registry.central'
    on the OAM interface, it must add the system controller OAM
    subnet to the subcloud override file in the dcmanager.

    Depends-On: https://review.opendev.org/#/c/690082/
    Change-Id: I9ee6f0f99a940d5d2cc2f245977ffb9f207916c5
    Partial-Bug: 1846799
    Signed-off-by: Tao Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692444

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692444
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=d4ef79e4985f4ea4da1e31c789b6fd02dcb7a94c
Submitter: Zuul
Branch: master

commit d4ef79e4985f4ea4da1e31c789b6fd02dcb7a94c
Author: Tao Liu <email address hidden>
Date: Thu Oct 31 15:36:26 2019 -0400

    Allow subcloud to use central-cloud's local registry

    Remove the IP address URL encoding as it is not required for the
    the dnsmasq.

    Change-Id: I25bd6d637e468188b4641c601c0d30e26f4c09d6
    Partial-Bug: 1846799
    Signed-off-by: Tao Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692612

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692613

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692634

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692611
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=d78cd10a52e07e0071efccf6566720f95f64725f
Submitter: Zuul
Branch: master

commit d78cd10a52e07e0071efccf6566720f95f64725f
Author: Angie Wang <email address hidden>
Date: Fri Nov 1 14:59:45 2019 -0400

    Update to push system images under the default registry namespaces

    Currently, if a system is installed with private registry, the
    image will be pushed to the local registry with the prefix of
    private registry url.

    For exmaple,
    k8s.gcr.io alternative registry => privateregistry.com/k8s.gcr.io

    k8s.gcr.io/kube-apiserver:v1.16.2
    => registry.local:9001/privateregistry.com/k8s.gcr.io/kube-apiserver:v1.16.2

    This commit updates to push system images to the original default
    registry namespace which doesn't include the private registry url.

    For example,
    k8s.gcr.io/kube-apiserver:v1.16.2
    => registry.local:9001/k8s.gcr.io/kube-apiserver:v1.16.2

    This update makes the system images download/pushing mechanism to
    align with the application images.

    Partial-Bug: 1846799
    Change-Id: If79c254dedbc7dab902364f76aa2109bb99474c7
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692613
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=f04cb41a9eeaca0bd9c9c4d3400ffda14a5117df
Submitter: Zuul
Branch: master

commit f04cb41a9eeaca0bd9c9c4d3400ffda14a5117df
Author: Angie Wang <email address hidden>
Date: Fri Nov 1 15:00:37 2019 -0400

    Update armada image reference

    In commit https://review.opendev.org/#/c/692611/,
    it changes all system images to push under the default registry
    namespaces.

    With/without private registries, controller should pull armada
    image from "registry.local:9001/quay.io"

    Closes-Bug: 1846799
    Depends-On: https://review.opendev.org/#/c/692611/1
    Change-Id: Icb1ba83bfc54b468b277c5745fbc0de1d395ef4f
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692612
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=b05885ec499093f7037fa8f4a7bca43b5c7cda30
Submitter: Zuul
Branch: master

commit b05885ec499093f7037fa8f4a7bca43b5c7cda30
Author: Angie Wang <email address hidden>
Date: Fri Nov 1 15:00:09 2019 -0400

    Update kubeadm template with registry info

    In commit https://review.opendev.org/#/c/692611/,
    it changes all system images to push under the default registry
    namespaces. Update the kubeadm template to refer to the default
    k8s registry which is "k8s.gcr.io".

    With/without private registries, standby controller and worker
    nodes should pull images from "registry.local:9001/k8s.gcr.io".

    Partial-Bug: 1846799
    Depends-On: https://review.opendev.org/#/c/692611/1
    Change-Id: I1363ea680a0232682a014e7517687ce925f28f3c
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692634
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=538a491cb6498f08cd6ae66c70f5a67bd3bbbb3f
Submitter: Zuul
Branch: master

commit 538a491cb6498f08cd6ae66c70f5a67bd3bbbb3f
Author: Angie Wang <email address hidden>
Date: Fri Nov 1 22:53:13 2019 -0400

    Update the image generation logic from playbook

    This commit removes the logic that updates image reference with
    docker registry if the registry is not in one of the override-able
    public registries.

    Since we should support to download additional images via the
    parameter additional_local_registry_images from a registry that
    is not in one of the public registries.

    Change-Id: I2e47e04de46c22c6deb7de6363e9262ac5eb047c
    Closes-Bug: 1846799
    Signed-off-by: Angie Wang <email address hidden>

[Zhang Kunpeng (zhangkunpeng)]

Anybody could tell me how to set bootstrap-values.yml when deploy subcloud with registry.cental?
Thanks!