R3.0.2.x build11 - setup_all fails as contrail-schema fails to start with contrail networking pkg

[Sudheendra Rao]

contrail-schema fails to start during fab setup_all using contrail-networking package on R3.0.2.x build11.

2016-11-14 16:44:04:689586: [root@10.204.217.172] out: ======= Starting the services ======
2016-11-14 16:44:04:689658: [root@10.204.217.172] out: + for svc in '$msg_svc' '$web_svc' memcached
2016-11-14 16:44:04:689731: [root@10.204.217.172] out: + service rabbitmq-server restart
2016-11-14 16:44:04:689803: [root@10.204.217.172] out: unix:///tmp/supervisord_support_service.sock no such file
2016-11-14 16:44:04:790741: [root@10.204.217.172] out: + for svc in '$msg_svc' '$web_svc' memcached
2016-11-14 16:44:04:790865: [root@10.204.217.172] out: + service apache2 restart
2016-11-14 16:44:04:790944: [root@10.204.217.172] out: apache2: unrecognized service
2016-11-14 16:44:04:791019: [root@10.204.217.172] out: + for svc in '$msg_svc' '$web_svc' memcached
2016-11-14 16:44:04:791093: [root@10.204.217.172] out: + service memcached restart
2016-11-14 16:44:04:791166: [root@10.204.217.172] out: memcached: unrecognized service
2016-11-14 16:44:04:791239: [root@10.204.217.172] out: + service neutron-server restart
2016-11-14 16:44:04:791312: [root@10.204.217.172] out: neutron-server stop/waiting
2016-11-14 16:44:04:807196: [root@10.204.217.172] out: neutron-server start/running, process 19891
2016-11-14 16:44:04:823986: [root@10.204.217.172] out:
2016-11-14 16:44:04:840031:
2016-11-14 16:44:04:844423: [root@10.204.217.172] Executing task 'verify_cfgm'
2016-11-14 16:44:04:844985: [root@10.204.217.172] sudo: service supervisor-config status
2016-11-14 16:44:04:845223: [root@10.204.217.172] out: supervisor-config start/running, process 19190
2016-11-14 16:44:04:911348: [root@10.204.217.172] out:
2016-11-14 16:44:04:911550:
2016-11-14 16:44:04:911848: [root@10.204.217.172] sudo: service contrail-api status
2016-11-14 16:44:04:912070: [root@10.204.217.172] out: contrail-api:0 RUNNING pid 19212, uptime 0:00:24
2016-11-14 16:44:05:053121: [root@10.204.217.172] out:
2016-11-14 16:44:05:060687:
2016-11-14 16:44:05:067685: [root@10.204.217.172] sudo: service contrail-discovery status
2016-11-14 16:44:05:067918: [root@10.204.217.172] out: contrail-discovery:0 RUNNING pid 19393, uptime 0:00:19
2016-11-14 16:44:05:209514: [root@10.204.217.172] out:
2016-11-14 16:44:05:217156:
2016-11-14 16:44:05:223585: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:44:05:223833: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:44:05:373042: [root@10.204.217.172] out:
2016-11-14 16:44:05:374673:
2016-11-14 16:44:05:378899: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:44:25:399503: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:44:25:513619: [root@10.204.217.172] out:
2016-11-14 16:44:25:521738:
2016-11-14 16:44:25:527213: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:44:45:547971: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:44:45:686330: [root@10.204.217.172] out:
2016-11-14 16:44:45:686563:
2016-11-14 16:44:45:686704: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:45:05:704055: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:45:05:825653: [root@10.204.217.172] out:
2016-11-14 16:45:05:825881:
2016-11-14 16:45:05:831655: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:45:25:856442: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:45:25:985384: [root@10.204.217.172] out:
2016-11-14 16:45:25:989995:
2016-11-14 16:45:26:001889: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:45:46:013920: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:45:46:128545: [root@10.204.217.172] out:
2016-11-14 16:45:46:136151:
2016-11-14 16:45:46:143273: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:46:06:152227: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:46:06:366040: [root@10.204.217.172] out:
2016-11-14 16:46:06:366266:
2016-11-14 16:46:06:366604: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:46:26:385599: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:46:26:522646: [root@10.204.217.172] out:
2016-11-14 16:46:26:522837:
2016-11-14 16:46:26:524469: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:46:46:537179: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:46:46:650346: [root@10.204.217.172] out:
2016-11-14 16:46:46:657828:
2016-11-14 16:46:46:664603: [root@10.204.217.172] sudo: service contrail-schema status
2016-11-14 16:47:06:677480: [root@10.204.217.172] out: contrail-schema FATAL Exited too quickly (process log may have details)
2016-11-14 16:47:06:813744: [root@10.204.217.172] out:
2016-11-14 16:47:06:813989:

The setup details are:
ontrol Nodes : [u'nodei13-vm2', u'nodei14-vm2']
Compute Nodes : [u'nodea12', u'nodea15', u'nodea17']
Openstack Node : [u'nodek7-vm2']
WebUI Node : nodei13-vm2
Analytics Nodes : [u'nodei13-vm2']
Database Nodes : [u'nodei14-vm2']
Physical Devices : [u'hooper', u"'hooper'"]

The ip-address of the VMs are
nodei13-vm2: 10.204.217.172
nodei14-vm2: 10.204.217.179
nodek7-vm2: 10.204.216.111

Schema log stack-trace:
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/bin/contrail-schema in <module>()
    6
    7 if __name__ == '__main__':
    8 sys.exit(
    9 load_entry_point('schema-transformer==0.1dev', 'console_scripts', 'contrail-schema')()
   10 )
load_entry_point = <function load_entry_point>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in server_main()
  842 def server_main():
  843 cgitb.enable(format='text')
  844 main()
  845 # end server_main
  846
global main = <function main>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in main(args_str='--conf_file /etc/contrail/contrail-schema.conf -... --conf_file /etc/contrail/contrail-database.conf')
  836 _zookeeper_client.master_election(zk_path_pfx + "/schema-transformer",
  837 os.getpid(), run_schema_transformer,
  838 args)
  839 # end main
  840
args = Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181')

 /usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py in master_election(self=<cfgm_common.zkclient.ZookeeperClient object>, path='/schema-transformer', identifier=19768, func=<function run_schema_transformer>, *args=(Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181'),), **kwargs={})
  335 def master_election(self, path, identifier, func, *args, **kwargs):
  336 self._election = self._zk_client.Election(path, identifier)
  337 self._election.run(func, *args, **kwargs)
  338 # end master_election
  339
self = <cfgm_common.zkclient.ZookeeperClient object>
self._election = <kazoo.recipe.election.Election object>
self._election.run = <bound method Election.run of <kazoo.recipe.election.Election object>>
func = <function run_schema_transformer>
args = (Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181'),)
kwargs = {}

 /usr/lib/python2.7/dist-packages/kazoo/recipe/election.py in run(self=<kazoo.recipe.election.Election object>, func=<function run_schema_transformer>, *args=(Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181'),), **kwargs={})
   46 try:
   47 with self.lock:
   48 func(*args, **kwargs)
   49
   50 except CancelledError:
func = <function run_schema_transformer>
args = (Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181'),)
kwargs = {}

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in run_schema_transformer(args=Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181'))
  805 _vnc_lib = VncApi(
  806 args.admin_user, args.admin_password, args.admin_tenant_name,
  807 args.api_server_ip, args.api_server_port, api_server_use_ssl=args.api_server_use_ssl)
  808 connected = True
  809 connection_state_update(ConnectionStatus.UP)
args = Namespace(admin_password='contrail123', admin_te..._ip='10.204.217.179:2181', zk_server_port='2181')
args.api_server_ip = '10.204.217.172'
args.api_server_port = '8082'
api_server_use_ssl undefined
args.api_server_use_ssl = 'False'

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in __init__(self=<vnc_api.vnc_api.VncApi object>, username='admin', password='contrail123', tenant_name='admin', api_server_host='10.204.217.172', api_server_port='8082', api_server_url=None, conf_file=None, user_info=None, auth_token=None, auth_host=None, auth_port=None, auth_protocol=None, auth_url=None, auth_type=None, wait_for_connect=False, api_server_use_ssl='False', domain_name=None)
  320 try:
  321 homepage = self._request(rest.OP_GET, self._base_url,
  322 retry_on_error=False)
  323 self._parse_homepage(homepage)
  324 except ServiceUnavailableError as e:
retry_on_error undefined
builtinFalse = False

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in _request(self=<vnc_api.vnc_api.VncApi object>, op=2, url='/', data=None, retry_on_error=False, retry_after_authn=False, retry_count=30)
  746 self._headers = self._authenticate(content, self._headers)
  747 # Recursive call after authentication (max 1 level)
  748 content = self._request(op, url, data=data, retry_after_authn=True)
  749
  750 return content
content = u'Authentication required'
self = <vnc_api.vnc_api.VncApi object>
self._request = <bound method VncApi._request of <vnc_api.vnc_api.VncApi object>>
op = 2
url = '/'
data = None
retry_after_authn = False
builtinTrue = True

 /usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py in _request(self=<vnc_api.vnc_api.VncApi object>, op=2, url='/', data=None, retry_on_error=True, retry_after_authn=True, retry_count=30)
  771 raise BadRequest(status, content)
  772 else: # Unknown Error
  773 raise HttpError(status, content)
  774 # end while True
  775
global HttpError = <class 'cfgm_common.exceptions.HttpError'>
status = 401
content = u'Authentication required'
<class 'cfgm_common.exceptions.HttpError'>: HTTP Status: 401 Content: Authentication required
    __class__ = <class 'cfgm_common.exceptions.HttpError'>
    __delattr__ = <method-wrapper '__delattr__' of HttpError object>
    __dict__ = {'content': u'Authentication required', 'status_code': 401}
    __doc__ = None
    __format__ = <built-in method __format__ of HttpError object>
    __getattribute__ = <method-wrapper '__getattribute__' of HttpError object>
    __getitem__ = <method-wrapper '__getitem__' of HttpError object>
    __getslice__ = <method-wrapper '__getslice__' of HttpError object>
    __hash__ = <method-wrapper '__hash__' of HttpError object>
    __init__ = <bound method HttpError.__init__ of HttpError()>
    __module__ = 'cfgm_common.exceptions'
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of HttpError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of HttpError object>
    __repr__ = <method-wrapper '__repr__' of HttpError object>
    __setattr__ = <method-wrapper '__setattr__' of HttpError object>
    __setstate__ = <built-in method __setstate__ of HttpError object>
    __sizeof__ = <built-in method __sizeof__ of HttpError object>
    __str__ = <bound method HttpError.__str__ of HttpError()>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of HttpError object>
    __weakref__ = None
    args = ()
    content = u'Authentication required'
    message = ''
    status_code = 401

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/bin/contrail-schema", line 9, in <module>
    load_entry_point('schema-transformer==0.1dev', 'console_scripts', 'contrail-schema')()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 844, in server_main
    main()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 838, in main
    args)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py", line 337, in master_election
    self._election.run(func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kazoo/recipe/election.py", line 48, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 807, in run_schema_transformer
    args.api_server_ip, args.api_server_port, api_server_use_ssl=args.api_server_use_ssl)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 322, in __init__
    retry_on_error=False)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 748, in _request
    content = self._request(op, url, data=data, retry_after_authn=True)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 773, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 401 Content: Authentication required