Kubernetes cluster is getting IPs from “Management Network” instead of "Cluster Host Network"

[Fabiano Correa Mercer]

Brief Description
-----------------
According to the StarlingX documentation (https://docs.starlingx.io/planning/kubernetes/the-cluster-host-network.html), All nodes in the cluster must be attached to the cluster host network.
But Nodes are using IPs from management network.

Severity
--------
Minor

Steps to Reproduce
------------------
configure an AIO-DX or controlers+worker scenario:
( https://docs.starlingx.io/deploy_install_guides/r7_release/virtual/controller_storage_install_kubernetes.html )

Defines the IPs of “Cluster-host subnet” and “Management subnet” respectively.

Included the same IPs in the Localhost.yml file

Bootstrap the Ansible-playbook using the Localhost.yml file
Check output of commands:
kubectl get nodes -A -o wide
kubectl get pod -A -o wide

Expected Behavior
------------------
Kubectl nodes should use IPs from Cluster-host subnet.

Check output of commands:
kubectl get nodes -A -o wide
kubectl get pod -A -o wide

Actual Behavior
----------------
After completion of Duplex installation Bootstrapping, it was analyzed that Controller-0 and Controller-1 were got IP from “management subnet” instead “Cluster-host subnet”

Check output of commands:
kubectl get nodes -A -o wide
kubectl get pod -A -o wide

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Two node system, Multi-node system
IPv4,IPv6

Branch/Pull Time/Commit
-----------------------
Probably introduced by:
https://review.opendev.org/q/topic:bug%252F1830779

Last Pass
---------
N/A

Timestamp/Logs
--------------
[root@controller-0 sysadmin(keystone_admin)]# system addrpool-list
 +------------------------+----------+
 | name | network |
 +------------------------+----------+
 | cluster-host-subnet | fd02:: |
 | management | fd01:: |
 +------------------------+----------+

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get nodes -A -o wide
 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
 compute-0 Ready <none> 14h v1.24.4 fd01::7fe6:c32:4561:fd1e <none>
 controller-0 Ready control-plane,master 16h v1.24.4 fd01::2 <none>
 controller-1 Ready control-plane,master 15h v1.24.4 fd01::3 <none>

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pod -A -o wide
 kube-system calico-node-6v4gg 1/1 Running fd01::3 controller-1
 kube-system calico-node-r5qhr 1/1 Running fd01::2 controller-0
 kube-system cephfs-nodeplugin-hzcdm 2/2 Running fd01::3 controller-1
 kube-system cephfs-nodeplugin-mgk94 2/2 Running fd01::2 controller-0

Test Activity
-------------
Evaluation

Workaround
----------
N/A

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/867737
Committed: https://opendev.org/starlingx/config/commit/891e71e2f2d59bd82c66ca7551534a2b9f300449
Submitter: "Zuul (22348)"
Branch: master

commit 891e71e2f2d59bd82c66ca7551534a2b9f300449
Author: Fabiano Mercer <email address hidden>
Date: Wed Dec 14 16:24:01 2022 -0300

    Attach all nodes to the cluster host network

    Problem: Nodes in the cluster are attached to the management
    network instead of the cluster host network.

    According to StarlingX documentation, all nodes in the cluster
    must be attached to the cluster host network.

    Test plan ( Debian only )
    PASS Installed 2+1 system using IPv4
         (2controllers, 1 worker)
    PASS Installed 2+1 system using IPv6
         (2controllers, 1 worker)
    PASS AIO-SX system using IPv4
    PASS AIO-SX system using IPv6
    PASS AIO-SX system external access to a Kubernetes service
    PASS AIO-DX system using IPv4 and tested traffic between PODs
    PASS AIO-DX system using IPv6 and tested traffic between PODs
    PASS AIO-SX upgrade
    PASS STANDARD ( 2 controllers + 1 worker ) upgrade
    PASS STANDARD ( 2 controllers + 1 worker ) pod -> external traffic

    Closes-Bug: #1999690

    Signed-off-by: Fabiano Mercer <email address hidden>
    Change-Id: Ib9ea1a857f657ca6fe193aab5cb692d3baf76c4b

[Ghada Khalil (gkhalil)]

Re-opening. The commit caused issues on worker nodes and was reverted: https://review.opendev.org/c/starlingx/config/+/869946

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/871393
Committed: https://opendev.org/starlingx/config/commit/be4dce95ffb271a4365b1b938f229691eaba3b62
Submitter: "Zuul (22348)"
Branch: master

commit be4dce95ffb271a4365b1b938f229691eaba3b62
Author: Fabiano Mercer <email address hidden>
Date: Fri Jan 20 22:00:38 2023 -0300

    Create cluster-host-ipAttach all nodes to the cluster host network

    Problem: Nodes in the cluster are attached to the management
    network instead of the cluster host network.

    According to StarlingX documentation, all nodes in the cluster
    must be attached to the cluster host network.
    It is necessary to create the cluster-host for worker or
    storage node before the node_ip is required and just update
    the interface_id field of addresses database when the node's
    interface is being created.

    Test plan ( Debian only )
    PASS Installed 2+1 system using IPv4
         (2controllers, 1 worker)
    PASS Installed 2+1 system using IPv6
         (2controllers, 1 worker)
    PASS AIO-SX system using IPv4
    PASS AIO-SX system using IPv6
    PASS AIO-SX system external access to a Kubernetes service
    PASS AIO-DX system using IPv4 and tested traffic between PODs
    PASS AIO-DX system using IPv6 and tested traffic between PODs
    PASS AIO-SX upgrade
    PASS STANDARD ( 2 controllers + 1 worker ) upgrade
    PASS STANDARD ( 2 controllers + 1 worker ) pod -> external traffic

    Closes-Bug: #1999690

    Signed-off-by: Fabiano Mercer <email address hidden>
    Change-Id: I5aee984269a9710285092d8bc5def0842eb06f9f