Storage: Provision fails in a multi-openstack node setup

[Jeya ganesh babu J]

Storage provision fails in a multi-openstack node setup. This was seen when provisioning the staging cluster by Thomas.

On 5/4/16, 11:16 PM, "Thomas Murray" <email address hidden> wrote:
we had a lot of issues provisioning storage in the juniper it staging
cluster with 3.0.2/34. you may want to have a look at the logs on
root@10.163.41.4 (c0ntrail123). most of the time i just re-ran the
setup_storage and it went past the previous issues. twice i had hung
kernels and had to reboot. this one is the last i am seeing:
2016-05-04 23:09:20:327116: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.14] out:
2016-05-04 23:09:20:327146: [root@10.163.254.4] out: [10.163.41.4] out:
2016-05-04 23:09:20:327182: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.1] download: /tmp/contrail-storage-stats.conf <-
/etc/init/contrail-storage-stats.conf
2016-05-04 23:09:20:342710: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.8] run: ls /etc/init/contrail-storage-stats.conf
2>/dev/null | wc -l
2016-05-04 23:09:20:342772: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.8] out: 1
2016-05-04 23:09:20:358300: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.8] out:
2016-05-04 23:09:20:358435: [root@10.163.254.4] out: [10.163.41.4] out:
2016-05-04 23:09:20:358463: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.12] run: ls /etc/init/contrail-storage-stats.conf
2>/dev/null | wc -l
2016-05-04 23:09:20:358493: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.12] out: 1
2016-05-04 23:09:20:365921: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.12] out:
2016-05-04 23:09:20:365985: [root@10.163.254.4] out: [10.163.41.4] out:
2016-05-04 23:09:20:366016: [root@10.163.254.4] out: [10.163.41.4] out:
[root@10.163.41.4] run: sudo openstack-config --set
       /etc/contrail/contrail-storage-nodemgr.conf DEFAULTS
disc_server_ip 10.163.41.241
2016-05-04 23:09:20:366045: [root@10.163.254.4] out: [10.163.41.4] out:
Traceback (most recent call last):
2016-05-04 23:09:20:466371: [root@10.163.254.4] out: [10.163.41.4] out:
File "/usr/bin/storage-fs-setup", line 9, in <module>
2016-05-04 23:09:20:466466: [root@10.163.254.4] out: [10.163.41.4] out:
   load_entry_point('ContrailProvisioning==0.1dev', 'console_scripts',
'storage-fs-setup')()
2016-05-04 23:09:20:467744: [root@10.163.254.4] out: [10.163.41.4] out:
File
"/usr/local/lib/python2.7/dist-packages/contrail_provisioning/storage/sto
ragefs/setup.py", line 3770, in main
2016-05-04 23:09:20:467817: [root@10.163.254.4] out: [10.163.41.4] out:
   SetupCeph(args_str)
2016-05-04 23:09:20:467851: [root@10.163.254.4] out: [10.163.41.4] out:
File
"/usr/local/lib/python2.7/dist-packages/contrail_provisioning/storage/sto
ragefs/setup.py", line 3660, in __init__
2016-05-04 23:09:20:467878: [root@10.163.254.4] out: [10.163.41.4] out:
   self.do_storage_setup()
2016-05-04 23:09:20:467906: [root@10.163.254.4] out: [10.163.41.4] out:
File
"/usr/local/lib/python2.7/dist-packages/contrail_provisioning/storage/sto
ragefs/setup.py", line 3584, in do_storage_setup
2016-05-04 23:09:20:467961: [root@10.163.254.4] out: [10.163.41.4] out:
   self.do_configure_stats_daemon()
2016-05-04 23:09:20:467989: [root@10.163.254.4] out: [10.163.41.4] out:
File
"/usr/local/lib/python2.7/dist-packages/contrail_provisioning/storage/sto
ragefs/setup.py", line 3090, in do_configure_stats_daemon
2016-05-04 23:09:20:468015: [root@10.163.254.4] out: [10.163.41.4] out:
   for os_entry in zip(self._args.os_hosts):
2016-05-04 23:09:20:468041: [root@10.163.254.4] out: [10.163.41.4] out:
AttributeError: 'Namespace' object has no attribute 'os_hosts'
2016-05-04 23:09:20:468066: [root@10.163.254.4] out: [10.163.41.4] out:
Exception in thread Thread-2 (most likely raised during interpreter
shutdown):Exception in thread Thread-4 (most likely raised during
interpreter shutdown):Exception in thread Thread-12 (most likely raised
during interpreter shutdown):
2016-05-04 23:09:20:976189: [root@10.163.254.4] out: [10.163.41.4] out:
2016-05-04 23:09:20:976283: [root@10.163.254.4] out:
2016-05-04 23:09:21:007897: [root@10.163.254.4] out:
2016-05-04 23:09:21:007989: [root@10.163.254.4] out: Fatal error: run()
received nonzero return code 1 while executing!
2016-05-04 23:09:21:008021: [root@10.163.254.4] out:
i have not debugged it yet.
we will need to build out an 80 node cluster soon so i have concerns on
the resiliency/repeatability. let me know if you see anything i should
change.
thanks,
‹tom

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19938
Submitter: Jeya ganesh babu (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19939
Submitter: Jeya ganesh babu (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19939
Committed: http://github.org/Juniper/contrail-provisioning/commit/7df1195d5f954a6166cffcb3e9dab689eb0b5dc2
Submitter: Zuul
Branch: master

commit 7df1195d5f954a6166cffcb3e9dab689eb0b5dc2
Author: Jeya ganesh babu J <email address hidden>
Date: Thu May 5 11:05:27 2016 -0700

Cache tier support/Storage provision fix

Closes-Bug: #1578753
Closes-Bug: #1579210
Added support for Ceph cache tier
Fixed issue in provision of stats daemon in multi-openstack
node setup. The variable provided in the provision script was
corrected.

Change-Id: I2a17f377f6ca2877562e8fabc77ec0c4d6f3eb87

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19938
Committed: http://github.org/Juniper/contrail-provisioning/commit/523fa782cb8664dadc346b03e29d1f84b400296d
Submitter: Zuul
Branch: R3.0

commit 523fa782cb8664dadc346b03e29d1f84b400296d
Author: Jeya ganesh babu J <email address hidden>
Date: Thu May 5 11:02:52 2016 -0700

Storage provision fix

Closes-Bug: #1578753
Fixed issue in provision of stats daemon in multi-openstack
node setup. The variable provided in the provision script was
corrected.

Change-Id: If93315b1282027066a3db7b12f1b26f4449e5ffb