Sachin,
	Atleast from the command, the rabbit cluster seems to be fine ?
Ajay, Prakash, 
	Any thoughts ? 

root@nodec22:~/github/mine4/contrail-test# rabbitmqctl cluster_status
Cluster status of node rabbit@nodec22 ...
[{nodes,[{disc,[rabbit@nodeh5,rabbit@nodeh4,rabbit@nodec22]}]},
 {running_nodes,[rabbit@nodeh4,rabbit@nodeh5,rabbit@nodec22]}]
...done.
root@nodec22:~/github/mine4/contrail-test#





On 7/19/14, 5:05 AM, "Sachin Bansal" <email address hidden> wrote:

>I see the following in the logs:
>
>07/19/2014 04:57:01 AM [nodec22:Schema:Config:0]: Exception Unknown id:
>Error: oper 1 url /fqname-to-id body {"fq_name": ["default-domain",
>"TestBasicVMVN4-45770645", "vn222"], "type": "virtual-network"} response
>Name [u'default-domain', u'TestBasicVMVN4-45770645', u'vn222'] not found
>while creating VirtualNetworkST for default-
>domain:TestBasicVMVN4-45770645:vn222
>
>
>This means that schema transformer received this message from ifmap, but
>it wasn't there when it queried api server. On further debugging, I found
>that API server is not draining rabbit queues:
>
>root@nodec22:~# rabbitmqctl list_queues
>Listing queues ...
>vnc_config.nodeh4-9100	950893
>vnc_config.nodeh5-9100	948882
>vnc_config.nodec22-9100	928005
>
>
>Rabbitmq logs show this error:
>
>=ERROR REPORT==== 18-Jul-2014::14:06:21 ===
>Mnesia(rabbit@nodec22): ** ERROR ** mnesia_event got
>{inconsistent_database, starting_partitioned_network, rabbit@nodeh5}
>
>
>We will have to recluster rabbitmq.
>
>-- 
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1344233
>
>Title:
>  schema in a continuous loop and crashing
>
>Status in Juniper Openstack distribution:
>  New
>
>Bug description:
>  Build 1.10 2241
>  env.roledefs = {
>      'all': [host1, host2, host3, host4, host5],
>      'cfgm': [host1,host4,host5],
>      'openstack': [host2],
>      'control': [host1,host4],
>      'compute': [host3],
>      'collector': [host1,host4,host5],
>      'webui': [host1],    'database': [host1,host4,host5],
>      'build': [host_build],
>  }
>
>  env.hostnames = {
>      'all': ['nodec22', 'nodeg30', 'nodeg29', 'nodeh4', 'nodeh5']
>  }
>
>  On this setup, i had a few test scripts which ran in parallel.
>  There were 7-8 Vms which i deleted. After that schema on nodec22 is
>stuck some loop adding ACL, SG entries and it is repeatedly crashing as
>well. 
>
>  Logs of nodec22 will be in http://10.204.216.50/Docs/bugs/#
>
>  -----
>  schema.err: 
>
>  <type 'exceptions.AttributeError'>
>  Python 2.7.3: /usr/bin/python
>  Fri Jul 18 23:52:23 2014
>
>  A problem occurred in a Python script.  Here is the sequence of
>  function calls leading up to the error, in the order they occurred.
>
>   /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer
>object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
>   3366         result = arc_mapc.call('poll', pollreq)
>   3367         try:
>   3368             transformer.process_poll_result(result)
>   3369         except Exception as e:
>   3370             cgitb.Hook(
>  transformer = <schema_transformer.to_bgp.SchemaTransformer object>
>  transformer.process_poll_result = <bound method
>SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransfor
>mer object>>
>  result = '<?xml version="1.0" encoding="UTF-8"
>standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'
>
>   /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer
>object>, poll_result_str='<?xml version="1.0" encoding="UTF-8"
>standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
>   3023                     pass
>   3024                 else:
>   3025                     func(idents, meta)
>   3026             # end for meta
>   3027         # end for result_type
>  func = <bound method
>SchemaTransformer.add_virtual_netw...ema_transformer.to_bgp.SchemaTransfor
>mer object>>
>  idents = {'network-ipam':
>'default-domain:default-project:default-network-ipam', 'virtual-network':
>'default-domain:TestBasicVMVN2-56418410:vn2228'}
>  meta = <Element 
>{http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-ipam>
>
>   /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>add_virtual_network_network_ipam(self=<schema_transformer.to_bgp.SchemaTra
>nsformer object>, idents={'network-ipam':
>'default-domain:default-project:default-network-ipam', 'virtual-network':
>'default-domain:TestBasicVMVN2-56418410:vn2228'}, meta=<Element
>{http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-ipam>)
>   2706         subnet = VnSubnetsType()
>   2707         subnet.build(meta)
>   2708         virtual_network.ipams[ipam_name] = subnet
>   2709         virtual_network.update_ipams()
>   2710     # end add_virtual_network_network_ipam
>  virtual_network = None
>  virtual_network.ipams undefined
>  ipam_name = 'default-domain:default-project:default-network-ipam'
>  subnet = <vnc_api.gen.resource_xsd.VnSubnetsType object>
>  <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
>'ipams'
>      __class__ = <type 'exceptions.AttributeError'>
>      __delattr__ = <method-wrapper '__delattr__' of
>exceptions.AttributeError object>
>      __dict__ = {}
>      __doc__ = 'Attribute not found.'
>      __format__ = <built-in method __format__ of
>exceptions.AttributeError object>
>      __getattribute__ = <method-wrapper '__getattribute__' of
>exceptions.AttributeError object>
>      __getitem__ = <method-wrapper '__getitem__' of
>exceptions.AttributeError object>
>      __getslice__ = <method-wrapper '__getslice__' of
>exceptions.AttributeError object>
>      __hash__ = <method-wrapper '__hash__' of exceptions.AttributeError
>object>
>      __init__ = <method-wrapper '__init__' of exceptions.AttributeError
>object>
>      __new__ = <built-in method __new__ of type object>
>      __reduce__ = <built-in method __reduce__ of
>exceptions.AttributeError object>
>      __reduce_ex__ = <built-in method __reduce_ex__ of
>exceptions.AttributeError object>
>      __repr__ = <method-wrapper '__repr__' of exceptions.AttributeError
>object>
>      __setattr__ = <method-wrapper '__setattr__' of
>exceptions.AttributeError object>
>      __setstate__ = <built-in method __setstate__ of
>exceptions.AttributeError object>
>      __sizeof__ = <built-in method __sizeof__ of
>exceptions.AttributeError object>
>      __str__ = <method-wrapper '__str__' of exceptions.AttributeError
>object>
>      __subclasshook__ = <built-in method __subclasshook__ of type object>
>      __unicode__ = <built-in method __unicode__ of
>exceptions.AttributeError object>
>      args = ("'NoneType' object has no attribute 'ipams'",)
>      message = "'NoneType' object has no attribute 'ipams'"
>
>  The above is a description of an error in a Python program.  Here is
>  the original traceback:
>
>  Traceback (most recent call last):
>    File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py",
>line 3368, in launch_arc
>      transformer.process_poll_result(result)
>    File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py",
>line 3025, in process_poll_result
>      func(idents, meta)
>    File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py",
>line 2708, in add_virtual_network_network_ipam
>      virtual_network.ipams[ipam_name] = subnet
>  AttributeError: 'NoneType' object has no attribute 'ipams'
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/juniperopenstack/+bug/1344233/+subscriptions