Juniper Openstack

R3.0 vcenter-compute: Setup_all failed in openstack-HA setup

Bug #1532927 reported by shajuvk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Invalid
Medium
Andra Cismaru
Trunk
Invalid
Medium
Andra Cismaru
Juniper Openstack r3.1.0.0-fcs

Bug Description

Sanity log:

http://anamika.englab.juniper.net:8080/job/ubuntu-14-04_Juno_vcenter_compute_HA_Sanity/17/consoleFull

Setup_vcenter plugin failed on one of the cfgm:
==============================================

2016-01-10 22:58:55:431449: [root@10.84.13.33] out: [localhost] local: sudo vcenter-plugin-setup.sh
2016-01-10 22:58:55:435018: [root@10.84.13.33] out: contrail-vcenter-plugin: stopped
2016-01-10 22:59:07:451956: [root@10.84.13.33] out: error: <class 'httplib.BadStatusLine'>, '': file: /usr/lib/python2.7/httplib.py line: 379
2016-01-10 22:59:07:852962: [root@10.84.13.33] out:
2016-01-10 22:59:07:853136: [root@10.84.13.33] out: Fatal error: local() encountered an error (return code 2) while executing 'sudo vcenter-plugin-setup.sh'
2016-01-10 22:59:07:853223: [root@10.84.13.33] out:
2016-01-10 22:59:07:853306: [root@10.84.13.33] out: Aborting.
2016-01-10 22:59:07:853421: [root@10.84.13.33] out:
2016-01-10 22:59:07:853903:

2016-01-10 22:59:07:857566: Fatal error: sudo() received nonzero return code 1 while executing!
2016-01-10 22:59:07:857566:
2016-01-10 22:59:07:857566: Requested: setup-vcenter-plugin --vcenter_url 10.84.22.104 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter A6s32-VC-Compute-Sanity --vcenter_dvswitch dvswitch-1 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 10.84.13.201 --api_port 8082 --zookeeper_serverlist 10.84.13.32:2181,10.84.13.33:2181,10.84.13.38:2181 --vcenter_mode vcenter-as-compute --keystone_ip 10.84.13.201 --keystone_admin_user admin --keystone_admin_passwd contrail123 --keystone_admin_tenant_name admin --keystone_auth_protocol http --keystone_auth_port 35357
2016-01-10 22:59:07:857566: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vcenter-plugin --vcenter_url 10.84.22.104 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter A6s32-VC-Compute-Sanity --vcenter_dvswitch dvswitch-1 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 10.84.13.201 --api_port 8082 --zookeeper_serverlist 10.84.13.32:2181,10.84.13.33:2181,10.84.13.38:2181 --vcenter_mode vcenter-as-compute --keystone_ip 10.84.13.201 --keystone_admin_user admin --keystone_admin_passwd contrail123 --keystone_admin_tenant_name admin --keystone_auth_protocol http --keystone_auth_port 35357"
2016-01-10 22:59:07:857566:
2016-01-10 22:59:07:857616: Aborting.

plugin-stdout
=========
0 [main] INFO net.juniper.contrail.vcenter.VCenterMonitor - Config params vcenter url: https://10.84.22.104/sdk, _vcenterUsername: <email address hidden>, api server: 10.84.13.201
84 [main] INFO net.juniper.contrail.sandesh.VCenterHttpServer - HTTP server on port 8234 started.
96 [main] INFO net.juniper.contrail.zklibrary.MasterSelection - Creating zookepper client with connection string: 10.84.13.32:2181,10.84.13.33:2181,10.84.13.38:2181 timeoutvalue=1s latchpath=/vcenter-plugin and id=node-vcenter-plugin
:$
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:463)
        at sun.nio.ch.Net.bind(Net.java:455)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:100)
        at sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50)
        at sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35)
        at com.sun.net.httpserver.HttpServer.create(HttpServer.java:129)
        at net.juniper.contrail.sandesh.VCenterHttpServer.<init>(VCenterHttpServer.java:42)
        at net.juniper.contrail.sandesh.VCenterHttpProvider.<init>(VCenterHttpProvider.java:9)
        at net.juniper.contrail.sandesh.VCenterHttpServices.init(VCenterHttpServices.java:47)
        at net.juniper.contrail.vcenter.VCenterMonitor.main(VCenterMonitor.java:162)
67 [main] ERROR net.juniper.contrail.sandesh.VCenterHttpServer - Cannot start HTTP server on port 8234
91 [main] INFO net.juniper.contrail.zklibrary.MasterSelection - Creating zookepper client with connection string: 10.84.13.32:2181,10.84.13.33:2181,10.84.13.38:2181 timeoutvalue=1s latchpath=/vcenter-plugin and id=node-vcenter-plugin
6 [main] WARN org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (2147483647). Pinning to 29
359 [main] INFO net.juniper.contrail.vcenter.VCenterMonitor - Waiting for zookeeper Mastership ..
241 [main] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting

=========

plugin.log
——————

2016-01-11 11:52:43,319 - INFO [Thread-4:ApiConnectorImpl@231] - >> Request: GET, /projects
2016-01-11 11:52:43,320 - INFO [Thread-4:ApiConnectorImpl@257] - << Response Status: HTTP/1.0 503 Service Unavailable
2016-01-11 11:52:43,320 - WARN [Thread-4:ApiConnectorImpl@525] - list failed with :Service Unavailable
2016-01-11 11:52:43,320 - ERROR [Thread-4:VncDB@162] - ApiServer not fully awake yet.. retry again..
2016-01-11 11:52:43,320 - ERROR [Thread-4:VCenterNotify@564] - Waiting for API server before starting sync
2016-01-11 11:52:45,775 - ERROR [pool-1-thread-1:ContrailVRouterApi@79] - /10.84.13.222:9090 Create TTransportException: java.net.ConnectException: Connection refused
2016-01-11 11:52:45,775 - ERROR [pool-1-thread-1:ContrailVRouterApi@214] - /10.84.13.222:9090 PeriodicConnectionCheck: FAILED
2016-01-11 11:52:45,776 - ERROR [pool-1-thread-1:ContrailVRouterApi@79] - /10.84.13.223:9090 Create TTransportException: java.net.ConnectException: Connection refused
2016-01-11 11:52:45,777 - ERROR [pool-1-thread-1:ContrailVRouterApi@214] - /10.84.13.223:9090 PeriodicConnectionCheck: FAILED
2016-01-11 11:52:48,321 - INFO [Thread-4:VncDB@157] - Checking if api-server is alive and kicking..
2016-01-11 11:52:48,321 - INFO [Thread-4:ApiConnectorImpl@163] - http connection <10.84.13.201, 8082> does not exit
2016-01-11 11:52:48,322 - INFO [Thread-4:ApiConnectorImpl@167] - http connection <10.84.13.201, 8082> established
2016-01-11 11:52:48,322 - INFO [Thread-4:ApiConnectorImpl@231] - >> Request: GET, /projects
2016-01-11 11:52:48,323 - INFO [Thread-4:ApiConnectorImpl@257] - << Response Status: HTTP/1.0 503 Service Unavailable
2016-01-11 11:52:48,323 - WARN [Thread-4:ApiConnectorImpl@525] - list failed with :Service Unavailable
2016-01-11 11:52:48,324 - ERROR [Thread-4:VncDB@162] - ApiServer not fully awake yet.. retry again..
2016-01-11 11:52:48,324 - ERROR [Thread-4:VCenterNotify@564] - Waiting for API server before starting sync
2016-01-11 11:52:49,971 - INFO [main:VCenterMonitor@157] - Config params vcenter url: https://10.84.22.104/sdk, _vcenterUsername: <email address hidden>, api server: 10.84.13.201
2016-01-11 11:52:50,014 - ERROR [main:VCenterHttpServer@46] - Cannot start HTTP server on port 8234
2016-01-11 11:52:50,015 - ERROR [main:VCenterHttpServer@51] - Cannot start HTTP server on port 8234
2016-01-11 11:52:50,032 - INFO [main:MasterSelection@47] - Creating zookepper client with connection string: 10.84.13.32:2181,10.84.13.33:2181,10.84.13.38:2181 timeoutvalue=1s latchpath=/vcenter-plugin and id=node-vcenter-plugin
2016-01-11 11:52:50,357 - INFO [main:VCenterMonitor@166] - Waiting for zookeeper Mastership ..
2016-01-11 11:52:52,493 - INFO [main:MasterSelection@71] - Awaiting Leadership....
2016-01-11 11:52:53,324 - INFO [Thread-4:VncDB@157] - Checking if api-server is alive and kicking..
2016-01-11 11:52:53,325 - INFO [Thread-4:ApiConnectorImpl@163] - http connection <10.84.13.201, 8082> does not exit
2016-01-11 11:52:53,325 - INFO [Thread-4:ApiConnectorImpl@167] - http connection <10.84.13.201, 8082> established
2016-01-11 11:52:53,325 - INFO [Thread-4:ApiConnectorImpl@231] - >> Request: GET, /projects

==================

zookeeper.log:
===========

root@a6s33:/var/log/zookeeper# tail -f zookeeper.log
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2016-01-11 11:54:47,005 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /10.84.13.33:48433 which had sessionid 0x2522f6c
450c1044
2016-01-11 11:54:48,329 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x2522f6c450c1043, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2016-01-11 11:54:48,330 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /10.84.13.33:48431 which had sessionid 0x2522f6c
450c1043
2016-01-11 11:54:50,000 - INFO [SessionTracker:ZooKeeperServer@325] - Expiring session 0x2522f6c450c1044, timeout of 4000ms exceeded
2016-01-11 11:54:50,000 - INFO [SessionTracker:ZooKeeperServer@325] - Expiring session 0x1522f6c44ad0f90, timeout of 120000ms exceeded
2016-01-11 11:54:50,001 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x2522f6c450c1044
2016-01-11 11:54:50,001 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x1522f6c44ad0f90
2016-01-11 11:54:52,000 - INFO [SessionTracker:ZooKeeperServer@325] - Expiring session 0x2522f6c450c1036, timeout of 120000ms exceeded
2016-01-11 11:54:52,000 - INFO [SessionTracker:ZooKeeperServer@325] - Expiring session 0x1522f6c44ad0fbc, timeout of 120000ms exceeded
2016-01-11 11:54:52,000 - INFO [SessionTracker:ZooKeeperServer@325] - Expiring session 0x1522f6c44ad0fbd, timeout of 120000ms exceeded
2016-01-11 11:54:52,000 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x2522f6c450c1036
2016-01-11 11:54:52,001 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x1522f6c44ad0fbc
2016-01-11 11:54:52,001 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x1522f6c44ad0fbd
2016-01-11 11:55:00,526 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.84.13.33:49198
2016-01-11 11:55:00,527 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /10.84.13.33:49198
2016-01-11 11:55:00,602 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.84.13.33:49206
2016-01-11 11:55:00,611 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /10.84.13.33:49206

===

root@a6s33:/var/log/contrail# tail -f contrail-vcenter-plugin.log
2016-01-11 11:24:46,009 - ERROR [main:VCenterHttpServer@51] - Cannot start HTTP server on port 8234
2016-01-11 11:24:46,024 - INFO [main:MasterSelection@47] - Creating zookepper client with connection string: 10.84.13.32:2181,10.84.13.33:2181,10.84.13.38:2181 timeoutvalue=1s latchpath=/vcenter-plugin and id=node-vcenter-plugin
2016-01-11 11:24:46,445 - INFO [main:VCenterMonitor@166] - Waiting for zookeeper Mastership ..
2016-01-11 11:24:48,630 - INFO [main:MasterSelection@71] - Awaiting Leadership....
2016-01-11 11:25:02,523 - INFO [main:VCenterMonitor@157] - Config params vcenter url: https://10.84.22.104/sdk, _vcenterUsername: <email address hidden>, api server: 10.84.13.201
2016-01-11 11:25:02,651 - ERROR [main:VCenterHttpServer@46] - Cannot start HTTP server on port 8234
2016-01-11 11:25:02,652 - ERROR [main:VCenterHttpServer@51] - Cannot start HTTP server on port 8234
2016-01-11 11:25:02,685 - INFO [main:MasterSelection@47

root@a6s33:/var/log/contrail#
root@a6s33:/var/log/contrail# netstat -nalp | grep 8234
tcp6 0 0 :::8234 :::* LISTEN 19484/java
root@a6s33:/var/log/contrail#

====

root@a6s33:/var/log/contrail# ps -eaf | grep 19484
root 9502 19774 0 11:25 pts/10 00:00:00 grep --color=auto 19484
contrail 19484 19483 0 Jan10 ? 00:00:31 java -cp ./juniper-contrail-vcenter.jar:./juniper-contrail-api.jar:./juniper-contrail-vijava.jar:./juniper-contrail-vrouter-api.jar:./lib/commons-exec-1.2.jar:./lib/commons-net-3.3.jar:./lib/commons-lang-2.6.jar:./lib/dom4j-1.6.1.jar:./lib/guava-14.0.jar:./lib/gson-1.7.1.jar:./lib/httpclient-4.2.1.jar:./lib/httpcore-4.2.1.jar:./lib/libthrift-0.8.0.jar:./lib/log4j-1.2.16.jar:./lib/slf4j-api-1.5.8.jar:./lib/slf4j-simple-1.5.8.jar:./lib/curator-framework-2.7.0.jar:./lib/curator-client-2.7.0.jar:./lib/curator-recipes-2.7.0.jar:./lib/openstack4j-2.0.1-withdeps.jar:./lib/openstack4j-jersey2-2.0.1.jar:./lib/openstack4j-core-2.0.1.jar:/usr/share/java/zookeeper.jar -Dlog4j.configuration=file:./log4j.properties net.juniper.contrail.vcenter.VCenterMonitor

See original description
Tags: sanity vmware
shajuvk (shajuvk)
description: updated
tags: added: sanity vmware
information type: Proprietary → Public
Revision history for this message
shajuvk (shajuvk) wrote :

provisioning works on mainline 2712 build

Revision history for this message
Sachchidanand Vaidya (vaidyasd) wrote :

We have already added shutdown hook for Introspect server. Can you check if this is still and issue with Shaju's HA setup.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Bug update]

bug update...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.