Distributed Cloud: default IPv6 gateway missing resulting in comms failure between sys controller and subclouds

[Maria Yousaf]

Brief Description
-----------------
Default IPv6 gateway missing resulting in comms failure between sys controller and subclouds (subcloud reported as offline)

Severity
--------
Major

Steps to Reproduce
------------------
- The distributed cloud system controller was installed successfully
- The load was installed on the subcloud
- subcloud_config was run using the generated subcloud config
- After subcloud configuration completed, the subcloud was shown as offline
- Closer examination of the networking, showed the default route to be missing
- The following was done to establish communications resulting in the subcloud coming online:

[wrsroot@controller-0 ~(keystone_admin)]$ sudo /sbin/route -A inet6 add default gw fd01:2::1
sudo: ldap_sasl_bind_s(): Can't contact LDAP server

[wrsroot@controller-0 ~(keystone_admin)]$ /sbin/ip -6 route show
fd01:2::/64 dev ens785f0.127 proto kernel metric 256 pref medium
fe80::/64 dev eno1 proto kernel metric 256 pref medium
fe80::/64 dev ens785f0 proto kernel metric 256 pref medium
fe80::/64 dev ens785f0.127 proto kernel metric 256 pref medium
default via fd01:2::1 dev ens785f0.127 metric 1 pref medium

[wrsroot@controller-0 ~(keystone_admin)]$ ping6 fd01:2::1
PING fd01:2::1(fd01:2::1) 56 data bytes
64 bytes from fd01:2::1: icmp_seq=1 ttl=64 time=1.51 ms
64 bytes from fd01:2::1: icmp_seq=2 ttl=64 time=1.52 ms
64 bytes from fd01:2::1: icmp_seq=3 ttl=64 time=0.715 ms
^C
--- fd01:2::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.715/1.252/1.529/0.379 ms
[wrsroot@controller-0 ~(keystone_admin)]$ ping6 fd01:1::1
PING fd01:1::1(fd01:1::1) 56 data bytes
64 bytes from fd01:1::1: icmp_seq=1 ttl=64 time=0.760 ms
64 bytes from fd01:1::1: icmp_seq=2 ttl=64 time=1.00 ms
^C
--- fd01:1::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.760/0.880/1.001/0.124 ms

The following is the permanent change:
sudo vim /etc/sysconfig/network-scripts/route6-ens785f0.127

Add:
default via fd01:2::1 dev ens785f0.127

It should be noted that the subcloud config file did contain the gateway information as follows:

[REGION2_PXEBOOT_NETWORK]
PXEBOOT_CIDR = 192.168.202.0/24
[MGMT_NETWORK]
VLAN = 127
CIDR = fd01:2::0/64
GATEWAY = fd01:2::1
<snip>

Expected Behavior
------------------
User should not need to provision default gateway manually as this information is present in the subcloud config file

Actual Behavior
----------------
Manual provisioning of routes was required

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed cloud system using IPv6

Branch/Pull Time/Commit
-----------------------
stx.10.2018 as of 2018-10-15_01-52-00

Timestamp/Logs
--------------
N/A

[Ghada Khalil (gkhalil)]

Required for stx.2018.10 as this impacts DC provisioning

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/612458

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612458
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=34b83ba9259f60b84ce9f40003645b50caf72f11
Submitter: Zuul
Branch: master

commit 34b83ba9259f60b84ce9f40003645b50caf72f11
Author: Teresa Ho <email address hidden>
Date: Mon Oct 22 13:45:21 2018 -0400

    Create route to system controller for mgmt vlan

    This update corrects the problem that the route to the system
    controller is not created when management vlan interface is
    created.

    Change-Id: Id6445e7667bbd3771c480970900bf0d4c1942bb0
    Closes-Bug: 1798609
    Signed-off-by: Teresa Ho <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/2018.10
Review: https://review.openstack.org/612501

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612501
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=a62219e6acb02a3b90f214dea775209b582ee887
Submitter: Zuul
Branch: r/2018.10

commit a62219e6acb02a3b90f214dea775209b582ee887
Author: Teresa Ho <email address hidden>
Date: Mon Oct 22 13:45:21 2018 -0400

    Create route to system controller for mgmt vlan

    This update corrects the problem that the route to the system
    controller is not created when management vlan interface is
    created.

    Change-Id: Id6445e7667bbd3771c480970900bf0d4c1942bb0
    Closes-Bug: 1798609
    Signed-off-by: Teresa Ho <email address hidden>
    (cherry picked from commit 34b83ba9259f60b84ce9f40003645b50caf72f11)