SM:R3.1build33: Provisioning gets stuck with mongo connection error for database node in a MultiInterface Setup

[sundarkh]

SM:R3.1build33: Provisioning gets stuck with mongo connection error for database node in a MultiInterface Setup

1) Install R3.1 build 33 Kilo Ubuntu SM
2) reimage the target with ubuntu 14.04.04. reimaged successfully
3) Add the contrail package ubuntu R3.1 build 33 kilo, Initiate Provision. Observe that the provision gets stuck at haproxy completed, and not proceeding with database role cluster with following error

Sep 13 23:39:02 nodec33 puppet-agent[1268]: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find resource 'Notify[contrail::profile::mongodb - {primary_db_ip} = 192.168.100.3]' for relationship on 'Exec[exec_mongo_connection]' on node nodec33.englab.juniper.net

Setup details :

SM : 10.204.217.160

root@nodej5:~# server-manager status server
{
    "server": [
        {
            "id": "nodec57",
            "ip_address": "10.204.221.61",
            "mac_address": "00:25:90:C5:58:6E",
            "status": "provision_started"
        },
        {
            "id": "nodec33",
            "ip_address": "10.204.221.59",
            "mac_address": "00:25:90:C4:82:28",
            "status": "haproxy_completed"
        },
        {
            "id": "nodec35",
            "ip_address": "10.204.221.58",
            "mac_address": "00:25:90:C4:7A:70",
            "status": "haproxy_completed"
        },
        {
            "id": "nodea4",
            "ip_address": "10.204.221.60",
            "mac_address": "00:25:90:A5:3B:12",
            "status": "provision_started"
        }

root@nodej5:~# server-manager show server --select id,cluster_id,roles,ip_address
{
    "server": [
        {
            "cluster_id": "cluster_multi",
            "id": "nodec57",
            "ip_address": "10.204.221.61",
            "roles": [
                "compute"
            ]
        },
        {
            "cluster_id": "cluster_multi",
            "id": "nodec33",
            "ip_address": "10.204.221.59",
            "roles": [
                "webui",
                "config",
                "database",
                "control",
                "collector"
            ]
        },
        {
            "cluster_id": "cluster_multi",
            "id": "nodec35",
            "ip_address": "10.204.221.58",
            "roles": [
                "control",
                "config",
                "openstack"
            ]
        },
        {
            "cluster_id": "cluster_multi",
            "id": "nodea4",
            "ip_address": "10.204.221.60",
            "roles": [
                "compute"
            ]
        }
    ]
}

Notes :

1) Issue seen with only Ubuntu Multi Interface setup; (tested with kilo, mitaka)
2) Issue not seen in Ubuntu Single node, Open stack HA setups;
3) Seen in only ubuntu target. Centos Multi interface provisioning goes through fine

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24137
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24138
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24137
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24138
Committed: http://github.org/Juniper/contrail-puppet/commit/6b110295b6a8f983a98c76710f31cd77ae7bbf49
Submitter: Zuul
Branch: R3.1

commit 6b110295b6a8f983a98c76710f31cd77ae7bbf49
Author: nitishkrishna <email address hidden>
Date: Wed Sep 14 10:05:00 2016 -0700

Closes-Bug: #1623374 - Dependency on notify left hanging after notify was removed

This case was not tested because it is seen only in ceilometer enabled case
Patch2:
Changed dependency to whatever was before the notify

Change-Id: I8a3c20d352f76a2984b138451eb4398ea63f94ed

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24137
Committed: http://github.org/Juniper/contrail-puppet/commit/a34de2bd51f09fcbbb2d3df27e9d73e85d3b3436
Submitter: Zuul
Branch: master

commit a34de2bd51f09fcbbb2d3df27e9d73e85d3b3436
Author: nitishkrishna <email address hidden>
Date: Wed Sep 14 10:05:00 2016 -0700

Closes-Bug: #1623374 - Dependency on notify left hanging after notify was removed

This case was not tested because it is seen only in ceilometer enabled case
Patch2:
Changed dependency to whatever was before the notify
Change-Id: I8a3c20d352f76a2984b138451eb4398ea63f94ed