Vcenter-only: fresh install fails on build#18 @ setup-vnc-collector with roles played by same controller nodes
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.1 |
New
|
Critical
|
Raj Reddy | |||
Trunk |
New
|
Critical
|
Raj Reddy |
Bug Description
This is regression failure and it worked fine with build#14
Provisioning roles are as below,
env.roledefs = {
'all': [host2, host3, host4, esxi5, esxi6, esxi8, esxi10, esxi12],
'cfgm': [host2, host3, host4],
'control': [host2, host3, host4],
'compute': [esxi5, esxi6, esxi8, esxi10, esxi12],
'collector': [host2, host3, host4],
'webui': [host2],
'database': [host2, host3, host4],
'build': [host_build],
'storage-
'storage-
# 'vgw': [host4, host5], # Optional, Only to enable VGW. Only compute can support vgw
# 'backup'
}
>>>>>>>>>> fab setup_all fails
::
::
2016-08-08 11:57:06:476600: [root@10.87.26.199] out: sed: can't read /etc/default/
2016-08-08 11:57:06:477831: [root@10.87.26.199] out:
2016-08-08 11:57:06:477916: [root@10.87.26.199] out: Warning: local() encountered an error (return code 2) while executing 'sed -i 's/crashkernel=
2016-08-08 11:57:06:477968: [root@10.87.26.199] out:
2016-08-08 11:57:06:478018: [root@10.87.26.199] out: [localhost] local: [ -f /etc/default/
2016-08-08 11:57:06:478065: [root@10.87.26.199] out:
2016-08-08 11:57:06:479400: [root@10.87.26.199] out: Warning: local() encountered an error (return code 1) while executing '[ -f /etc/default/
2016-08-08 11:57:06:479485: [root@10.87.26.199] out:
2016-08-08 11:57:06:479538: [root@10.87.26.199] out: [localhost] local: update-grub
2016-08-08 11:57:06:479585: [root@10.87.26.199] out: Generating grub configuration file ...
2016-08-08 11:57:07:581000: [root@10.87.26.199] out: Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
2016-08-08 11:57:08:181838: [root@10.87.26.199] out: Found linux image: /boot/vmlinuz-
2016-08-08 11:57:08:282187: [root@10.87.26.199] out: Found initrd image: /boot/initrd.
2016-08-08 11:57:08:282301: [root@10.87.26.199] out: done
2016-08-08 11:57:09:046737: [root@10.87.26.199] out: [localhost] local: sudo mv /tmp/tmpqgE12l/
2016-08-08 11:57:09:046926: [root@10.87.26.199] out: [localhost] local: sudo mv /tmp/tmpqgE12l/
2016-08-08 11:57:09:054343: [root@10.87.26.199] out: [localhost] local: sed -i 's/secret "secret123";/secret "xvysmOR8lnUQRB
2016-08-08 11:57:09:061734: [root@10.87.26.199] out: [localhost] local: sed -i 's/secret "secret123";/secret "xvysmOR8lnUQRB
2016-08-08 11:57:09:061820: [root@10.87.26.199] out: [localhost] local: sudo mv /tmp/tmpqgE12l/
2016-08-08 11:57:09:063085: [root@10.87.26.199] out: [localhost] local: sudo control-
2016-08-08 11:57:09:070479: [root@10.87.26.199] out: stop: Unknown instance:
2016-08-08 11:57:09:134066: [root@10.87.26.199] out: supervisor-control start/running, process 33150
2016-08-08 11:57:09:134148: [root@10.87.26.199] out:
2016-08-08 11:57:09:149595:
2016-08-08 11:57:09:149828: [root@10.87.26.199] sudo: python -c 'from platform import linux_distribution; print linux_distribut
2016-08-08 11:57:09:149977: [root@10.87.26.199] out: ('Ubuntu', '14.04', 'trusty')
2016-08-08 11:57:09:214660: [root@10.87.26.199] out:
2016-08-08 11:57:09:214824:
2016-08-08 11:57:09:214971: [root@10.87.26.197] Executing task 'verify_control'
2016-08-08 11:57:09:215274: [root@10.87.26.197] sudo: service supervisor-control status
2016-08-08 11:57:09:215407: [root@10.87.26.197] out: supervisor-control start/running, process 14943
2016-08-08 11:57:09:234870: [root@10.87.26.197] out:
2016-08-08 11:57:09:235017:
2016-08-08 11:57:09:239653: [root@10.87.26.197] sudo: service contrail-control status
2016-08-08 11:57:09:239804: [root@10.87.26.197] out: contrail-control RUNNING pid 15123, uptime 0:00:07
2016-08-08 11:57:09:405298: [root@10.87.26.197] out:
2016-08-08 11:57:09:412950:
2016-08-08 11:57:09:416332: [root@10.87.26.208] Executing task 'verify_control'
2016-08-08 11:57:09:416497: [root@10.87.26.208] sudo: service supervisor-control status
2016-08-08 11:57:09:416645: [root@10.87.26.208] out: supervisor-control start/running, process 32263
2016-08-08 11:57:09:451103: [root@10.87.26.208] out:
2016-08-08 11:57:09:451243:
2016-08-08 11:57:09:451466: [root@10.87.26.208] sudo: service contrail-control status
2016-08-08 11:57:09:451612: [root@10.87.26.208] out: contrail-control STARTING
2016-08-08 11:57:09:585398: [root@10.87.26.208] out:
2016-08-08 11:57:09:585527:
2016-08-08 11:57:09:586504: [root@10.87.26.208] sudo: service contrail-control status
2016-08-08 11:57:29:606533: [root@10.87.26.208] out: contrail-control RUNNING pid 32320, uptime 0:00:23
2016-08-08 11:57:29:710147: [root@10.87.26.208] out:
2016-08-08 11:57:29:710297:
2016-08-08 11:57:29:711379: [root@10.87.26.199] Executing task 'verify_control'
2016-08-08 11:57:29:711518: [root@10.87.26.199] sudo: service supervisor-control status
2016-08-08 11:57:29:711694: [root@10.87.26.199] out: supervisor-control start/running, process 33150
2016-08-08 11:57:29:730286: [root@10.87.26.199] out:
2016-08-08 11:57:29:730455:
2016-08-08 11:57:29:735238: [root@10.87.26.199] sudo: service contrail-control status
2016-08-08 11:57:29:735390: [root@10.87.26.199] out: contrail-control RUNNING pid 33160, uptime 0:00:19
2016-08-08 11:57:29:868659: [root@10.87.26.199] out:
2016-08-08 11:57:29:868928:
2016-08-08 11:57:29:869120: [root@10.87.26.197] Executing task 'setup_collector'
2016-08-08 11:57:29:869301: [root@10.87.26.197] sudo: python -c 'from platform import linux_distribution; print linux_distribut
2016-08-08 11:57:29:869563: [root@10.87.26.197] out: ('Ubuntu', '14.04', 'trusty')
2016-08-08 11:57:29:903980: [root@10.87.26.197] out:
2016-08-08 11:57:29:904305:
2016-08-08 11:57:29:904411: [root@10.87.26.197] sudo: service redis-server stop
2016-08-08 11:57:29:904585: [root@10.87.26.197] out: Stopping redis-server: redis-server.
2016-08-08 11:57:30:286385: [root@10.87.26.197] out:
2016-08-08 11:57:31:287525:
2016-08-08 11:57:31:287775: [root@10.87.26.197] sudo: sed -i -e '/^[ ]*bind/s/^/#/' /etc/redis/
2016-08-08 11:57:31:287990: [root@10.87.26.197] sudo: sed -i -e 's/lua-
2016-08-08 11:57:31:337607: [root@10.87.26.197] sudo: grep '^dbfilename' /etc/redis/
2016-08-08 11:57:31:387113: [root@10.87.26.197] out: dump.rdb
2016-08-08 11:57:31:467715: [root@10.87.26.197] out:
2016-08-08 11:57:31:467849:
2016-08-08 11:57:31:468020: [root@10.87.26.197] sudo: grep '^dir' /etc/redis/
2016-08-08 11:57:31:468128: [root@10.87.26.197] out: /var/lib/redis
2016-08-08 11:57:31:549273: [root@10.87.26.197] out:
2016-08-08 11:57:31:549427:
2016-08-08 11:57:31:549619: [root@10.87.26.197] sudo: rm -f /var/lib/
2016-08-08 11:57:31:549733: [root@10.87.26.197] sudo: sed -i -e '/^[ ]*save/s/^/#/' /etc/redis/
2016-08-08 11:57:31:598677: [root@10.87.26.197] sudo: sed -i -e '/^[ ]*dbfilename/
2016-08-08 11:57:31:647034: [root@10.87.26.197] sudo: service redis-server start
2016-08-08 11:57:31:728067: [root@10.87.26.197] out: Starting redis-server: redis-server.
2016-08-08 11:57:31:812003: [root@10.87.26.197] out:
2016-08-08 11:57:31:812208:
2016-08-08 11:57:31:825359: [root@10.87.26.197] sudo: service redis-server status | grep not
2016-08-08 11:57:31:879721: Warning: sudo() received nonzero return code 1 while executing 'service redis-server status | grep not'!
2016-08-08 11:57:31:879721:
2016-08-08 11:57:31:879721: 2016-08-08 11:57:31:825496: [root@10.87.26.197] sudo: python -c 'from platform import linux_distribution; print linux_distribut
2016-08-08 11:57:31:880310: [root@10.87.26.197] out: ('Ubuntu', '14.04', 'trusty')
2016-08-08 11:57:31:944781: [root@10.87.26.197] out:
2016-08-08 11:57:31:944965:
2016-08-08 11:57:31:945102: [root@10.87.26.197] sudo: rm /etc/init/
2016-08-08 11:57:31:945299: setup-vnc-collector --cassandra_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_port 5672 --cfgm_ip 172.16.80.2 --self_collector_ip 172.16.80.2 --num_nodes 3 --kafka_enabled True --aaa_mode no-auth
2016-08-08 11:57:31:994359: [root@10.87.26.197] sudo: setup-vnc-collector --cassandra_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_port 5672 --cfgm_ip 172.16.80.2 --self_collector_ip 172.16.80.2 --num_nodes 3 --kafka_enabled True --aaa_mode no-auth
2016-08-08 11:57:31:994474: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-08-08 11:57:32:128439: [root@10.87.26.197] out: [localhost] local: sudo sed 's/SELINUX=
2016-08-08 11:57:32:128658: [root@10.87.26.197] out: [localhost] local: sudo mv config.new /etc/selinux/config
2016-08-08 11:57:32:128749: [root@10.87.26.197] out: [localhost] local: setenforce 0
2016-08-08 11:57:32:136329: [root@10.87.26.197] out: /bin/sh: 1: setenforce: not found
2016-08-08 11:57:32:136438: [root@10.87.26.197] out:
2016-08-08 11:57:32:136512: [root@10.87.26.197] out: Warning: local() encountered an error (return code 127) while executing 'setenforce 0'
2016-08-08 11:57:32:136853: [root@10.87.26.197] out:
2016-08-08 11:57:32:136934: [root@10.87.26.197] out: [localhost] local: rm config.new
2016-08-08 11:57:32:136996: [root@10.87.26.197] out: rm: cannot remove ‘config.new’: No such file or directory
2016-08-08 11:57:32:137060: [root@10.87.26.197] out:
2016-08-08 11:57:32:137404: [root@10.87.26.197] out: Warning: local() encountered an error (return code 1) while executing 'rm config.new'
2016-08-08 11:57:32:137479: [root@10.87.26.197] out:
2016-08-08 11:57:32:137544: [root@10.87.26.197] out: [localhost] local: sudo chkconfig iptables off
2016-08-08 11:57:32:137606: [root@10.87.26.197] out: [localhost] local: sudo iptables --flush
2016-08-08 11:57:32:145070: [root@10.87.26.197] out: [localhost] local: sudo sed '/DAEMON_
2016-08-08 11:57:32:148405: [root@10.87.26.197] out: [localhost] local: sudo mv /etc/sysconfig/
2016-08-08 11:57:32:155898: [root@10.87.26.197] out: kernel.core_pattern = /var/crashes/
2016-08-08 11:57:32:163305: [root@10.87.26.197] out: [localhost] local: grep -q 'kernel.
2016-08-08 11:57:32:163443: [root@10.87.26.197] out: [localhost] local: sudo sed 's/net.
2016-08-08 11:57:32:163528: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/sysctl.new /etc/sysctl.conf
2016-08-08 11:57:32:171085: [root@10.87.26.197] out: [localhost] local: rm /tmp/sysctl.new
2016-08-08 11:57:32:174437: [root@10.87.26.197] out: rm: cannot remove ‘/tmp/
2016-08-08 11:57:32:177813: [root@10.87.26.197] out:
2016-08-08 11:57:32:178051: [root@10.87.26.197] out: Warning: local() encountered an error (return code 1) while executing 'rm /tmp/sysctl.new'
2016-08-08 11:57:32:178117: [root@10.87.26.197] out:
2016-08-08 11:57:32:178190: [root@10.87.26.197] out: [localhost] local: sysctl -p
2016-08-08 11:57:32:178251: [root@10.87.26.197] out: fs.file-max = 165535
2016-08-08 11:57:32:178340: [root@10.87.26.197] out: kernel.core_pattern = /var/crashes/
2016-08-08 11:57:32:178405: [root@10.87.26.197] out: [localhost] local: mkdir -p /var/crashes
2016-08-08 11:57:32:178726: [root@10.87.26.197] out: [localhost] local: chmod 777 /var/crashes
2016-08-08 11:57:32:179960: [root@10.87.26.197] out: [localhost] local: sed -i 's/crashkernel=
2016-08-08 11:57:32:183431: [root@10.87.26.197] out: sed: can't read /etc/default/
2016-08-08 11:57:32:183548: [root@10.87.26.197] out:
2016-08-08 11:57:32:183618: [root@10.87.26.197] out: Warning: local() encountered an error (return code 2) while executing 'sed -i 's/crashkernel=
2016-08-08 11:57:32:183681: [root@10.87.26.197] out:
2016-08-08 11:57:32:183744: [root@10.87.26.197] out: [localhost] local: [ -f /etc/default/
2016-08-08 11:57:32:183820: [root@10.87.26.197] out:
2016-08-08 11:57:32:185211: [root@10.87.26.197] out: Warning: local() encountered an error (return code 1) while executing '[ -f /etc/default/
2016-08-08 11:57:32:185314: [root@10.87.26.197] out:
2016-08-08 11:57:32:185392: [root@10.87.26.197] out: [localhost] local: update-grub
2016-08-08 11:57:32:185458: [root@10.87.26.197] out: Generating grub configuration file ...
2016-08-08 11:57:32:886499: [root@10.87.26.197] out: Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
2016-08-08 11:57:33:488742: [root@10.87.26.197] out: Found linux image: /boot/vmlinuz-
2016-08-08 11:57:33:552421: [root@10.87.26.197] out: Found initrd image: /boot/initrd.
2016-08-08 11:57:33:559836: [root@10.87.26.197] out: done
2016-08-08 11:57:34:324160: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpvK3cEF/
2016-08-08 11:57:34:324271: [root@10.87.26.197] out: [localhost] local: openstack-config --get /etc/contrail/
2016-08-08 11:57:34:324327: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:356002: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpvK3cEF/
2016-08-08 11:57:34:371616: [root@10.87.26.197] out: [localhost] local: [ -f /etc/contrail/
2016-08-08 11:57:34:375039: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:375125: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:406671: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:422155: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:437679: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:469165: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:476577: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:508168: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:523662: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:555201: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:570694: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:602238: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:609778: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:641466: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:648873: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:680360: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:695953: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:727503: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:734990: [root@10.87.26.197] out: [localhost] local: openstack-config --get /etc/contrail/
2016-08-08 11:57:34:766546: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:781987: [root@10.87.26.197] out: [localhost] local: mkdir -p /etc/snmp
2016-08-08 11:57:34:797437: [root@10.87.26.197] out: [localhost] local: echo 'mibs +ALL' > /etc/snmp/snmp.conf
2016-08-08 11:57:34:800931: [root@10.87.26.197] out: [localhost] local: [ -f /etc/contrail/
2016-08-08 11:57:34:801193: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:802548: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:834092: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:849569: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:865008: [root@10.87.26.197] out: [localhost] local: [ -f /etc/contrail/
2016-08-08 11:57:34:896706: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:896818: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:928353: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:935797: [root@10.87.26.197] out: [localhost] local: contrail-config --set /etc/contrail/
2016-08-08 11:57:34:967329: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpvK3cEF/
2016-08-08 11:57:34:982833: [root@10.87.26.197] out: Traceback (most recent call last):
2016-08-08 11:57:34:986279: [root@10.87.26.197] out: File "/usr/bin/
2016-08-08 11:57:34:986361: [root@10.87.26.197] out: load_entry_
2016-08-08 11:57:34:986419: [root@10.87.26.197] out: File "/usr/local/
2016-08-08 11:57:34:986470: [root@10.87.26.197] out: collector.setup()
2016-08-08 11:57:34:987780: [root@10.87.26.197] out: File "/usr/local/
2016-08-08 11:57:34:987865: [root@10.87.26.197] out: self.fixup_
2016-08-08 11:57:34:995237: [root@10.87.26.197] out: File "/usr/local/
2016-08-08 11:57:34:995322: [root@10.87.26.197] out: self.fixup_
2016-08-08 11:57:34:995377: [root@10.87.26.197] out: File "/usr/local/
2016-08-08 11:57:34:995427: [root@10.87.26.197] out: '__keystone_
2016-08-08 11:57:34:995477: [root@10.87.26.197] out: AttributeError: 'Namespace' object has no attribute 'keystone_certfile'
2016-08-08 11:57:34:995525: [root@10.87.26.197] out:
2016-08-08 11:57:34:996892:
2016-08-08 11:57:35:032225: Fatal error: sudo() received nonzero return code 1 while executing!
2016-08-08 11:57:35:032225:
2016-08-08 11:57:35:032225: Requested: setup-vnc-collector --cassandra_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_port 5672 --cfgm_ip 172.16.80.2 --self_collector_ip 172.16.80.2 --num_nodes 3 --kafka_enabled True --aaa_mode no-auth
2016-08-08 11:57:35:032225: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vnc-collector --cassandra_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --zookeeper_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_ip_list 172.16.80.2 172.16.80.13 172.16.80.4 --amqp_port 5672 --cfgm_ip 172.16.80.2 --self_collector_ip 172.16.80.2 --num_nodes 3 --kafka_enabled True --aaa_mode no-auth"
2016-08-08 11:57:35:032225:
2016-08-08 11:57:35:032255: Aborting.
2016-08-08 11:57:35:032255: c0ntrail123
2016-08-08 11:57:35:032109: Disconnecting from 10.87.26.208... done.
2016-08-08 11:57:35:115032: Disconnecting from 10.87.26.197... done.
2016-08-08 11:57:35:196880: Disconnecting from 10.87.26.199... done.
nsarath@ ubuntu- build04: /auto/cores/ 1611087$ ubuntu- build04: /auto/cores/ 1611087$ pwd ubuntu- build04: /auto/cores/ 1611087$ ls -l ubuntu- build04: /auto/cores/ 1611087$
nsarath@
/auto/cores/1611087
nsarath@
total 304820
-rwxrwxrwx 1 nsarath test 117084160 Aug 8 12:23 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 4413440 Aug 8 12:23 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 87848960 Aug 8 12:23 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 101539840 Aug 8 12:24 Ctrl-C-log.tar
nsarath@