BGP peers are not getting provisioned on the control node using SMLite on mainline-3058

[Vivek krishnappa]

BGP Provisioning fails on Mitaka using sm-lite on mainline-3058.
No BGP peers are present post provisioning, and need to manually configure the BGP peer on the control node for the sessions to come up with external routers.
Provision was successful and no error messages were seen.

Logs::
From: “docker logs controller”:
TASK [contrail/control : register control node to config api server (keystone auth)] ***
ESC[0;33mchanged: [localhost]ESC[0m

TASK [contrail/control : register control node as a BGP speaker with md5 (keystone auth)] ***
ESC[0;36mskipping: [localhost]ESC[0m

TASK [contrail/control : register control node as a BGP speaker without md5 (keystone auth)] ***
ESC[0;33mchanged: [localhost]ESC[0m

TASK [contrail/control : Register external routers in bgp speaker list (keystone auth)] ***

TASK [redis : Run redis setup as per redis_setup_target variable] **************
ESC[0;36mincluded: /contrail-ansible-internal/playbooks/roles/redis/tasks/setup.yml for localhostESC[0m

From: /contrail-ansible-internal/playbooks/roles/contrail/control/tasks/provision/keystone.yml
- name: Wait till keystone answers
  uri:
    url: "{{ keystone_admin_auth_url }}"
    status_code: 200
  register: keystone_connect_status
  until: keystone_connect_status.status == 200
  retries: 1200
  delay: 1

- name: register control node to config api server (keystone auth)
  shell: "/usr/share/contrail-utils/provision_control.py --api_server_ip {{ api_server_ip }} --api_server_port {{ api_server_port }} --router_asn {{ bgp_asn }} {{ provision_cmd_keystone_args }}"

- name: register control node as a BGP speaker with md5 (keystone auth)
  shell: "/usr/share/contrail-utils/provision_control.py --api_server_ip {{ api_server_ip }} --api_server_port {{ api_server_port }} --host_name {{ ansible_hostname }} --host_ip {{ my_controller_ip }} --oper add --router_asn {{ bgp_asn }} --md5 {{ bgp_md5 }} {{ provision_cmd_keystone_args }}"
  when: bgp_md5 is defined

- name: register control node as a BGP speaker without md5 (keystone auth)
  shell: "/usr/share/contrail-utils/provision_control.py --api_server_ip {{ api_server_ip }} --api_server_port {{ api_server_port }} --host_name {{ ansible_hostname }} --host_ip {{ my_controller_ip }} --oper add --router_asn {{ bgp_asn }} {{ provision_cmd_keystone_args }}"
  when: bgp_md5 is not defined

- name: Register external routers in bgp speaker list (keystone auth)
  shell: "/usr/share/contrail-utils/provision_mx.py --api_server_ip {{ api_server_ip }} --api_server_port {{ api_server_port }} --router_name {{ item.key }} --router_ip {{ item.value }} --router_asn {{ bgp_asn }} {{ provision_cmd_keystone_args }}"
  with_dict: "{{ external_routers_list }}"

From External_router::
Apr 20 08:51:16 blr-mx1 rpd[60210]: bgp_recv: read from peer 10.204.216.48 (Internal AS 64512) failed: Connection reset by peer
Apr 20 08:54:04 blr-mx1 rpd[60210]: BGP_UNEXPECTED_MESSAGE_TYPE: bgp_read_message: peer 10.204.216.48 (Internal AS 64512): Notification arrived, expected Open
Apr 20 08:54:04 blr-mx1 rpd[60210]: bgp_read_message:2698: NOTIFICATION received from 10.204.216.48 (Internal AS 64512): code 6 (Cease) subcode 3 (Peer Unconfigured)
Apr 20 08:54:04 blr-mx1 rpd[60210]: bgp_recv: read from peer 10.204.216.48 (Internal AS 64512) failed: Connection reset by peer
Apr 20 08:56:52 blr-mx1 rpd[60210]: BGP_UNEXPECTED_MESSAGE_TYPE: bgp_read_message: peer 10.204.216.48 (Internal AS 64512): Notification arrived, expected Open
Apr 20 08:56:52 blr-mx1 rpd[60210]: bgp_read_message:2698: NOTIFICATION received from 10.204.216.48 (Internal AS 64512): code 6 (Cease) subcode 3 (Peer Unconfigured)
Apr 20 08:56:52 blr-mx1 rpd[60210]: bgp_recv: read from peer 10.204.216.48 (Internal AS 64512) failed: Connection reset by peer
Apr 20 08:59:40 blr-mx1 rpd[60210]: BGP_UNEXPECTED_MESSAGE_TYPE: bgp_read_message: peer 10.204.216.48 (Internal AS 64512): Notification arrived, expected Open
Apr 20 08:59:40 blr-mx1 rpd[60210]: bgp_read_message:2698: NOTIFICATION received from 10.204.216.48 (Internal AS 64512): code 6 (Cease) subcode 3 (Peer Unconfigured)
Apr 20 08:59:40 blr-mx1 rpd[60210]: bgp_recv: read from peer 10.204.216.48 (Internal AS 64512) failed: Connection reset by peer
Apr 20 09:02:28 blr-mx1 rpd[60210]: BGP_UNEXPECTED_MESSAGE_TYPE: bgp_read_message: peer 10.204.216.48 (Internal AS 64512): Notification arrived, expected Open
---(more 51%)---[abort]

[Vivek krishnappa (vivk)]

Provision was triggered using JSONs, with cluster json details as listed.
    "cluster" : [
        {
            "id": "cluster1",
            "parameters" : {
                "domain": "englab.juniper.net",
                "subnet_mask": "255.255.255.0",
                "gateway": "10.204.216.254",
                "provision": {
                    "contrail_4": {
                    },
                    "contrail": {
                        "config": {
                            "config_ip_list": ["10.204.216.48"]
                        },
                        "control": {
                            "encapsulation_priority": "VXLAN,MPLSoUDP,MPLSoGRE",
                            "router_asn": 64512,
                            "external_bgp": "[('mx1', '10.204.216.253')]"
                        }
                    },
                    "openstack":{
                        "openstack_manage_amqp": true,
                        "keystone": {
                            "admin_password": "contrail123",
                            "admin_token": "admin"
                        }
                    }
                }
            }
        }
    ]
}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30625
Submitter: Harish Kumar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30625
Committed: http://github.com/Juniper/contrail-docker/commit/a254cf7ce9ccd134b5695f6a9fff81d41b6f3671
Submitter: Zuul (<email address hidden>)
Branch: master

commit a254cf7ce9ccd134b5695f6a9fff81d41b6f3671
Author: hkumarmk <email address hidden>
Date: Thu Apr 20 21:44:47 2017 +0530

Update configuration samples and schema for mx routers

Change-Id: Ie06a5599821614b5bef6ca8145d14f896566c34d
Fixes-bug: 1684622

[Harish Kumar (hkumarmk)]

contrail-ansible to be updated to write /etc/contrailctl/controller.conf configuration as below

[CONTROLLER]
# external_routers_list - dictionary of external, physical routers in form of {name1: ip1, name2: ip2}
# external_routers_list = {'r1': '192.168.0.1', 'r2': '192.168.0.2', 'r3': '192.168.0.3'}

Assigning this to Abhay to update this

[Nitish Krishna Kaveri (nitishk)]

No code change is needed, the cluster json needs to have the following added:

“provision”: {
    “contrail_4”: {
         “controller_config”: {
              "external_routers_list": { “r1": “192.168.0.1", “r2": “192.168.0.2", “r3": “192.168.0.3”}
          }
    }
}

[Vivek krishnappa (vivk)]

After adding the said parameters to cluster json, the provisioning fails and the control docker keeps restarting.

root@nodea10:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f4f58bd8fb2c 10.204.216.48:5100/contrail-cloud-docker-pkg-contrail-analytics:mainline "/bin/sh -c /entry..." 12 minutes ago Up 3 minutes analytics
4fcbfa1a5a10 10.204.216.48:5100/contrail-cloud-docker-pkg-contrail-analyticsdb:mainline "/bin/sh -c /entry..." 12 minutes ago Up 38 seconds analyticsdb
cdc138c0bc57 10.204.216.48:5100/contrail-cloud-docker-pkg-contrail-controller:mainline "/bin/sh -c /entry..." 12 minutes ago Restarting (1) 5 minutes ago controller
3bffcd1a7546 registry:2 "/entrypoint.sh /e..." 40 minutes ago Up 12 minutes registry

Logs from docker control:
=====================================
Additional properties are not allowed ('external_routers_list' was unexpected)^M
+ set -e^M
+ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin^M
+ DAEMON=/usr/bin/supervisord^M
+ ANSIBLE_INVENTORY=all-in-one^M
+ test -x /usr/bin/supervisord^M
+ LOG=/var/log/supervisord.log^M
+ '[' -f /etc/default/supervisord ']'^M
+ DAEMON_OPTS='-n -c /etc/contrail/supervisord.conf '^M
+ trap cleanup SIGHUP SIGINT SIGTERM^M
+ ulimit -s unlimited^M
+ ulimit -c unlimited^M
+ ulimit -d unlimited^M
+ ulimit -v unlimited^M
+ ulimit -n 4096^M
+ contrailctl config sync -c controller -F -v -t configure^M
Additional properties are not allowed ('external_routers_list' was unexpected)^M

[Ramprakash R (ramprakash)]

@Vivek: The fix https://review.opencontrail.org/30625 is not in 3058 containers yet.
I checked that the controller.conf in your setup now has the required parameters in the required syntax. Please try this in the next build which has the above mentioned fix.

FYI: the relevant snippet contrailctl/controller.conf after you changed the JSON
%<----------------
root@nodea10:~# cat /etc/contrailctl/controller.conf
[CONTROL]
asn = 64512
bgp_port = 179
[GLOBAL]
....
....

[CONTROLLER]
external_routers_list = {'blr-xm1': '10.204.216.253'} <------ the new parameter as required.
encap_priority = VXLAN,MPLSoUDP,MPLSoGRE
[KEYSTONE]
ip = 10.204.216.48
admin_password = contrail123
----------------->%

[Abhay Joshi (abhayj)]

Build 3059 was toxic. We need to test this with next good build.

[Nitish Krishna Kaveri (nitishk)]

Ext_routers have to imported correctly from testbed.py to JSONs.
Working on this

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30902
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30902
Committed: http://github.com/Juniper/contrail-server-manager/commit/d57dfb1dd8da6a0e65c2ca520e305a5f13cb08cf
Submitter: Zuul (<email address hidden>)
Branch: master

commit d57dfb1dd8da6a0e65c2ca520e305a5f13cb08cf
Author: nitishkrishna <email address hidden>
Date: Mon May 1 13:21:27 2017 -0700

Closes-Bug: #1684622 Import ext_routers from testbed as external routers list

Param named changed from external_routers to external_routers_list and moved to controller_config

In testbed.py:
ext_routers = [('mx1', '10.204.216.253')]

In JSON:
"parameters": {
"provision": {
"contrail_4": {
"controller_config": {
"external_routers_list": {
"mx1": "10.204.216.253"
}
},

Also importing encap_priority and router ASN from testbed.py

Change-Id: Ifb7f8ed85ba828a3d159732cf346d9f74e54451c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31191
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[Nitish Krishna Kaveri (nitishk)]

Re-opened by mistake