SM: R3.1.1 build 38 :After reimage , nodes are unable to connect to the SM in multi interface setup

[sundarkh]

SM: After reimage , nodes are unable to connect to the SM in multi interface setup

1) Install R3.1.1 build 38 Liberty SM , add centos images in SM database
2) Create json files to create multi interface cluster
3) Issue reimage to all the nodes
4) Observe that the re image gets completed succesfully (Centos gets installed on the target nodes)
5) After reimage gets completed, targets are unable to ping the puppet
6) node which had issue to connect to puppet, had multiple entry for control data interface(route -n) Screen Shot attached

Workaround

1) The gateway entry in the /etc/sysconfig/network-scripts/ifcfg-interfacename was commented
2) Restarted the network service
3) route table is proper now, and the connection to puppet works

Notes
------
1)Issue not seen with node, which has bond interface as control data interface
2)Though the gateway is populated for control data interface in the config file for a ubuntu node, 3)route table is proper
4)Issue was not seen with build 37

Setup details
--------------
SM : 10.204.217.160
/root/sm_files/cluster_multi_inf_new_param_centos.json
/root/sm_files/server_multi_if_new_param_centos.json

[sundarkh (sundar-kh)]

Multiple entry for control data interface

[kamlesh parmar (kparmar)]

The problem is not seen in ubuntu because dhcp supplied gateway is preferred but same behavior is not seen in centos.
Please remove the default gateway from the private network, you can not use the default gateway in private network.
The bond interface case is working because default gateway is not being configured for bond interface at all.

We will add validation check to not allow multiple default gateways (per nic default gateway). This is low priority bug.

[Ritam Gangopadhyay (ritam)]

Seeing similar issue with ubuntu as well.
The computes have correct default route pointing to tha management n/w but on the nodes with other roles the default route points to 192.168.100.254 gw.

root@nodec28:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.100.254 0.0.0.0 UG 0 0 0 p1p2
10.204.217.0 * 255.255.255.0 U 0 0 0 p1p1
192.168.100.0 * 255.255.255.0 U 0 0 0 p1p2
root@nodec28:~#

This compromises the reachability of the target from other than the SM node. This blocks sanity as package download and installation fails due to absent ssh connectivity.

Whereas on the compute node the routes are fine:-

root@nodei17:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.204.217.254 0.0.0.0 UG 0 0 0 em1
10.204.217.0 * 255.255.255.0 U 0 0 0 em1
192.168.100.0 * 255.255.255.0 U 0 0 0 vhost0
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
root@nodei17:~#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25979
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25980
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25981
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25980
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25981
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25979
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25981
Committed: http://github.org/Juniper/contrail-server-manager/commit/4cc4f93ee8b66bb0955a3fdb83962d8e014d93e0
Submitter: Zuul
Branch: R3.1

commit 4cc4f93ee8b66bb0955a3fdb83962d8e014d93e0
Author: sgurumurthy <email address hidden>
Date: Thu Nov 10 21:18:01 2016 -0800

Closes-Bug: #1640064
Set the default gateway only for the management interface
gateway is no longer a mandatory reimage parameter

Change-Id: I67fb2469a8b3f7ed80bc5bbec4ba06dc442242e5

[sundarkh (sundar-kh)]

With Build 41,
---------------

After reimage ,

In a multi interface setup,

1) only management interface is created after reimage

root@nodec35:~# ifconfig -a
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1296 (1.2 KB) TX bytes:1296 (1.2 KB)

p1p1 Link encap:Ethernet HWaddr 00:25:90:c4:7a:70
          inet addr:10.204.221.58 Bcast:10.204.221.63 Mask:255.255.255.240
          inet6 addr: fe80::225:90ff:fec4:7a70/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:336 errors:0 dropped:0 overruns:0 frame:0
          TX packets:317 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:45486 (45.4 KB) TX bytes:46810 (46.8 KB)
          Memory:df980000-df9fffff

p1p2 Link encap:Ethernet HWaddr 00:25:90:c4:7a:71 >>>>> IP Address of control data network not there
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
          Memory:df900000-df97ffff

root@nodec35:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.204.221.62 0.0.0.0 UG 0 0 0 p1p1
10.204.221.48 0.0.0.0 255.255.255.240 U 0 0 0 p1p1

2) Both management and control interfaces are created after reimage for a node that has bond interface

nodea4:~# ifconfig
bond0 Link encap:Ethernet HWaddr 00:25:90:c4:98:a9
          inet addr:192.168.100.1 Bcast:192.168.100.255 Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fec4:98a9/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
          RX packets:8171 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6348 errors:0 dropped:2 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:506324 (506.3 KB) TX bytes:793224 (793.2 KB)

em1 Link encap:Ethernet HWaddr 00:25:90:c4:98:a9
          UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
          RX packets:64 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1587 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8064 (8.0 KB) TX bytes:203028 (203.0 KB)
          Interrupt:20 Memory:df900000-df920000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

p2p2 ...

[sundarkh (sundar-kh)]

In case of SM Lite scenario,

Provisioning fails with

{
    "return_code": 9,
    "return_data": null,
    "return_msg": "AddrFormatError('invalid IPNetwork '

Traceback

u'p')]"
"2016-11-12 14:29:12,862-DEBUG-server_mgr_main.py:3917-storage_get_control_network_mask(): STORAGE: control_ip: 192.168.100.3/24 => 192.168.100.0/24"
"2016-11-12 14:29:12,863-DEBUG-server_mgr_main.py:4124-build_calculated_cluster_params(): STORAGE: Control_network : 192.168.100.0/24"
"2016-11-12 14:29:12,865-ERROR-server_mgr_main.py:3358-log_trace(): *****TRACEBACK-START*****"
"2016-11-12 14:29:12,866-ERROR-server_mgr_main.py:3362-log_trace(): Traceback (most recent call last):
"
"2016-11-12 14:29:12,867-ERROR-server_mgr_main.py:3362-log_trace(): File "/opt/contrail/server_manager/server_mgr_main.py", line 4326, in provision_server
    server, cluster, role_servers, cluster_servers, package)
"
"2016-11-12 14:29:12,868-ERROR-server_mgr_main.py:3362-log_trace(): File "/opt/contrail/server_manager/server_mgr_main.py", line 4227, in build_calculated_provision_params
    server, cluster, role_servers, package)
"
"2016-11-12 14:29:12,869-ERROR-server_mgr_main.py:3362-log_trace(): File "/opt/contrail/server_manager/server_mgr_main.py", line 4189, in build_calculated_server_params
    server_control_gateway = self.get_control_gateway(server)
"
"2016-11-12 14:29:12,870-ERROR-server_mgr_main.py:3362-log_trace(): File "/opt/contrail/server_manager/server_mgr_main.py", line 3313, in get_control_gateway
    return str(IPNetwork(value['gateway']).ip)
"
"2016-11-12 14:29:12,870-ERROR-server_mgr_main.py:3362-log_trace(): File "/usr/lib/python2.7/dist-packages/netaddr/ip/__init__.py", line 917, in __init__
    else:
"
"2016-11-12 14:29:12,871-ERROR-server_mgr_main.py:3362-log_trace(): AddrFormatError: invalid IPNetwork
"

Testbed.py used

https://github.com/Juniper/contrail-tools/blob/master/testbeds/testbed_smgr_multi_interface.py.ubuntu-14.04

[sundarkh (sundar-kh)]

For SM Lite, https://bugs.launchpad.net/juniperopenstack/+bug/1641411 bug is raised

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25979
Committed: http://github.org/Juniper/contrail-server-manager/commit/6b0329cd2f1cb260cb310a62258be81c9fe1312b
Submitter: Zuul
Branch: master

commit 6b0329cd2f1cb260cb310a62258be81c9fe1312b
Author: sgurumurthy <email address hidden>
Date: Thu Nov 10 21:18:01 2016 -0800

Closes-Bug: #1640064
Set the default gateway only for the management interface
gateway is no longer a mandatory reimage parameter

Change-Id: I67fb2469a8b3f7ed80bc5bbec4ba06dc442242e5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25980
Committed: http://github.org/Juniper/contrail-server-manager/commit/2719e4a6ebf5c1a839554526a41d9d8b6b36690c
Submitter: Zuul
Branch: R3.2

commit 2719e4a6ebf5c1a839554526a41d9d8b6b36690c
Author: sgurumurthy <email address hidden>
Date: Thu Nov 10 21:18:01 2016 -0800

Closes-Bug: #1640064
Set the default gateway only for the management interface
gateway is no longer a mandatory reimage parameter

Change-Id: I67fb2469a8b3f7ed80bc5bbec4ba06dc442242e5

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26050
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26051
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26053
Submitter: sgurumurthy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26050
Committed: http://github.org/Juniper/contrail-server-manager/commit/8edb51939f74fb783c0f377d62c345f0694576c9
Submitter: Zuul
Branch: R3.1

commit 8edb51939f74fb783c0f377d62c345f0694576c9
Author: sgurumurthy <email address hidden>
Date: Mon Nov 14 13:40:25 2016 -0800

Closes-Bug: #1640064 Pass the macaddress to the interface_setup.py script if it is given as the inteface can
get renamed after reboot

Change-Id: I4f39dacbc4b88b4f512fba1323fd4fee86040699

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26053
Committed: http://github.org/Juniper/contrail-server-manager/commit/1a151ba539d35afde85fc34b7d582d6af735c49e
Submitter: Zuul
Branch: master

commit 1a151ba539d35afde85fc34b7d582d6af735c49e
Author: sgurumurthy <email address hidden>
Date: Mon Nov 14 13:40:25 2016 -0800

Closes-Bug: #1640064 Pass the macaddress to the interface_setup.py script if it is given as the inteface can
get renamed after reboot

Change-Id: I4f39dacbc4b88b4f512fba1323fd4fee86040699

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26051
Committed: http://github.org/Juniper/contrail-server-manager/commit/2af7057551a80d416007f332f403a39f84c8ac88
Submitter: Zuul
Branch: R3.2

commit 2af7057551a80d416007f332f403a39f84c8ac88
Author: sgurumurthy <email address hidden>
Date: Mon Nov 14 13:40:25 2016 -0800

Closes-Bug: #1640064 Pass the macaddress to the interface_setup.py script if it is given as the inteface can
get renamed after reboot

Change-Id: I4f39dacbc4b88b4f512fba1323fd4fee86040699