[R3.0-36]:docker provision failing during ceilometer setup

[alok kumar]

The docker provision failing during ceilometer setup.
http://anamika.englab.juniper.net:8080/job/Docker_ubuntu-14-04_kilo_Multi_Node_Sanity/48/console

Setup is locked.

Config Nodes : [u'nodei13-vm1', u'nodek7-vm1']
Control Nodes : [u'nodei13-vm1', u'nodek7-vm1']
Compute Nodes : [u'nodeg28', u'nodeg29', u'nodeg30']
Openstack Node : nodek7-vm1
WebUI Node : nodei14-vm1
Analytics Nodes : [u'nodei13-vm1', u'nodek7-vm1']
Physical Devices : [u"'hooper¹²]

Virtual node ip:
nodei13-vm1 - 10.204.217.103
nodei14-vm1 - 10.204.217.232
nodek7-vm1 - 10.204.216.110

sandip has done initial analysis:

Hi Megh

Setup s locked.

Now I see this
root@nodei14-vm1/opt/contrail/utils/fabfile# sudo mongo --hos
192.168.196.1 --quiet --eval 'rs.add("192.68.196.2:27017").ok'
0
Something iswrong.Could you check root@10.204.217.103

Regard
Sandip
On 5/18/16, 10:47 AM, "Sandip Dey" <email address hidden> wrote:

+Megh

Hi Megh>>
The mongodb cluster provisioning failed as

sudo: mongo --host 192.168.196.1 --quiet --eval
'rs.add("192.168.196.2:27017").ok’ returned ‘0’ , expected value is ‘1’.

When I ran the same command on the setup, it returned me ‘1’

root@nodei14-vm1:~# sudo mongo --host 192.168.196.1 --quiet --eval
'rs.add("19.168.196.2:27017").ok'
1

Could it be a timing issu?

I see in the code that , we are not retrying this command like the other
mongodb commans.Is there any specific reason that we are not retrying
this command?

# Add replicaSet members
        for other_ip in mongodb_ip_list:
           if ip == other_ip:
                continue
            cmd = "mongo --host " + ip + \
                " --quiet --eval 'rs.add(\"" + other_ip + ":27017\").ok'"
            verify_command_succeeded(cmd = cmd, expected_output = "1",
                                     error_str = "Not able to add " + \
                                         other_ip + " to replicaSet",
                                     max_count = 1, sleep_interval = 1,
                                     warn_only = False)

Regards
Sandip

[Megh Bhatt (meghb)]

We need to figure out the errmsg when rs.add() failed and then decide if a retry will work or not.
Please provide access to the setup and the commands to run to see this failure.

[Sudheendra Rao (sudheendra-k)]

this problem is not seen on latest builds, will provide required details if it's seen again.