1OS + 3CC topology provisioning broken

[Sarath]

Thilak debugged the setup and possible "keepalived" issue.

Taget-node logs
#####

root@a5d11e13:~#
root@a5d11e13:~#
root@a5d11e13:~# tail -f /var/log/syslog
Feb 2 17:06:05 a5d11e13 puppet-agent[7715]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Contrail::Keepalived::Keepalived[contrail_internal_ip] is already declared in file /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:71; cannot redeclare at /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:79 on node a5d11e13.englab.juniper.net
Feb 2 17:06:05 a5d11e13 puppet-agent[7715]: Not using cache on failed catalog
Feb 2 17:06:05 a5d11e13 puppet-agent[7715]: Could not retrieve catalog; skipping run
Feb 2 17:06:09 a5d11e13 puppet-agent[8403]: Local environment: "production" doesn't match server specified node environment "pvt_juno2709", switching agent to "pvt_juno2709".
Feb 2 17:06:13 a5d11e13 kernel: [15187.480030] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 2 17:06:14 a5d11e13 kernel: [15188.475874] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 2 17:06:15 a5d11e13 puppet-agent[8403]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Contrail::Keepalived::Keepalived[contrail_internal_ip] is already declared in file /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:71; cannot redeclare at /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:79 on node a5d11e13.englab.juniper.net
Feb 2 17:06:15 a5d11e13 puppet-agent[8403]: Not using cache on failed catalog
Feb 2 17:06:15 a5d11e13 puppet-agent[8403]: Could not retrieve catalog; skipping run
Feb 2 17:06:19 a5d11e13 puppet-agent[9385]: Local environment: "production" doesn't match server specified node environment "pvt_juno2709", switching agent to "pvt_juno2709".

Feb 2 17:06:23 a5d11e13 kernel: [15197.266210] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 2 17:06:24 a5d11e13 kernel: [15198.263170] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 2 17:06:25 a5d11e13 puppet-agent[9385]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Contrail::Keepalived::Keepalived[contrail_internal_ip] is already declared in file /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:71; cannot redeclare at /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:79 on node a5d11e13.englab.juniper.net
Feb 2 17:06:25 a5d11e13 puppet-agent[9385]: Not using cache on failed catalog
Feb 2 17:06:25 a5d11e13 puppet-agent[9385]: Could not retrieve catalog; skipping run
Feb 2 17:06:29 a5d11e13 puppet-agent[10128]: Local environment: "production" doesn't match server specified node environment "pvt_juno2709", switching agent to "pvt_juno2709".
Feb 2 17:06:33 a5d11e13 kernel: [15207.177935] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 2 17:06:34 a5d11e13 kernel: [15208.157453] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-vhost instead.
Feb 2 17:06:35 a5d11e13 puppet-agent[10128]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Contrail::Keepalived::Keepalived[contrail_internal_ip] is already declared in file /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:71; cannot redeclare at /etc/puppet/environments/pvt_juno2709/modules/contrail/manifests/keepalived.pp:79 on node a5d11e13.englab.juniper.net
Feb 2 17:06:35 a5d11e13 puppet-agent[10128]: Not using cache on failed catalog
Feb 2 17:06:35 a5d11e13 puppet-agent[10128]: Could not retrieve catalog; skipping run
^C
root@a5d11e13:~#
Broadcast message from root@a5d11e13
        (unknown) at 17:10 ...

Provisioning configs
###

}
root@Blaster-nsarath-004:~# server-manager show server --cluster_id issu-cluster --select "id,ip_address,roles,status"
{
    "server": [
        {
            "id": "a5d11e36",
            "ip_address": "10.87.143.84",
            "roles": [
                "openstack"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e14",
            "ip_address": "10.87.143.90",
            "roles": [],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e13",
            "ip_address": "10.87.143.92",
            "roles": [
                "config",
                "control",
                "collector",
                "webui",
                "database"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e12",
            "ip_address": "10.87.143.94",
            "roles": [
                "config",
                "control",
                "collector",
                "webui",
                "database"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e10",
            "ip_address": "10.87.143.98",
            "roles": [
                "config",
                "control",
                "collector",
                "webui",
                "database"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e09",
            "ip_address": "10.87.143.100",
            "roles": [
                "compute"
            ],
            "status": "reimage_completed"
        }
    ]
}
root@Blaster-nsarath-004:~#

root@Blaster-nsarath-004:~# dpkg -l | grep contrail
ii contrail-server-manager 3.0-2709 all Contrail Server Manager - Server Package
ii contrail-server-manager-client 3.0-2709 all Contrail Server Manager - Client Package
ii contrail-server-manager-cliff-client 3.0-2709 all Contrail Server Manager Cliff Client Package
ii contrail-server-manager-installer 3.0-2709~juno all Contrail Server Manager Installer Packages - Wrapper Package for Server Manager and Related Debian Packages
ii contrail-server-manager-monitoring 3.0-2709 all Contrail Server-manager Monitoring API Library package
ii contrail-web-core 3.0-2709 amd64 Contrail Systems Web UI
ii contrail-web-server-manager 3.0-2709 amd64 Contrail Systems Web UI Server Manager Feature
ii nodejs 0.10.35-1contrail1 amd64 Node.js event-based server-side javascript engine
ii python-backports.ssl-match-hostname 3.4.0.2-1contrail1 all The ssl.match_hostname() function from Python 3.4
ii python-certifi 1.0.1-1contrail1 all Python SSL Certificates
ii python-consistent-hash 1.0-0contrail1 amd64 <insert up to 60 chars description>
ii python-contrail 3.0-2709 amd64 OpenContrail python-libs
ii python-geventhttpclient 1.1.0-1contrail1 amd64 http client library for gevent
ii python-kazoo 1.3.1-1contrail2 all higher level API to Apache Zookeeper (Python 2)
ii python-pycassa 1.11.0-1contrail2 all Client library for Apache Cassandra
ii python-xmltodict 0.9.0-1contrail1 all Makes working with XML feel like you are working with JSON
root@Blaster-nsarath-004:~#

[Sarath (nsarath)]

>>> Please find below the logs on Target Node and Server-manager

-bash-4.1$
-bash-4.1$ hostname
centos-build02
-bash-4.1$
-bash-4.1$ pwd
/users/nsarath/PR/PR-1541164
-bash-4.1$
-bash-4.1$ ls -l
total 1076736
-rwxrwxrwx 1 nsarath test 36474880 Feb 2 17:45 5-log.tar*
-rwxrwxrwx 1 nsarath test 1061570560 Feb 2 17:46 SMlog.tar*
-rwxrwxrwx 1 nsarath test 194560 Feb 2 17:46 SMRunlog.tar*
-bash-4.1$

[Thilak Raj (tsurendra)]

Fixed in this bug https://bugs.launchpad.net/juniperopenstack/+bug/1540775

[Thilak Raj (tsurendra)]

https://review.opencontrail.org/16848