Comment 2 for bug 1344233

Sachin,
 Atleast from the command, the rabbit cluster seems to be fine ?
Ajay, Prakash,
 Any thoughts ?

root@nodec22:~/github/mine4/contrail-test# rabbitmqctl cluster_status
Cluster status of node rabbit@nodec22 ...
[{nodes,[{disc,[rabbit@nodeh5,rabbit@nodeh4,rabbit@nodec22]}]},
 {running_nodes,[rabbit@nodeh4,rabbit@nodeh5,rabbit@nodec22]}]
...done.
root@nodec22:~/github/mine4/contrail-test#

On 7/19/14, 5:05 AM, "Sachin Bansal" <email address hidden> wrote:

>I see the following in the logs:
>
>07/19/2014 04:57:01 AM [nodec22:Schema:Config:0]: Exception Unknown id:
>Error: oper 1 url /fqname-to-id body {"fq_name": ["default-domain",
>"TestBasicVMVN4-45770645", "vn222"], "type": "virtual-network"} response
>Name [u'default-domain', u'TestBasicVMVN4-45770645', u'vn222'] not found
>while creating VirtualNetworkST for default-
>domain:TestBasicVMVN4-45770645:vn222
>
>
>This means that schema transformer received this message from ifmap, but
>it wasn't there when it queried api server. On further debugging, I found
>that API server is not draining rabbit queues:
>
>root@nodec22:~# rabbitmqctl list_queues
>Listing queues ...
>vnc_config.nodeh4-9100 950893
>vnc_config.nodeh5-9100 948882
>vnc_config.nodec22-9100 928005
>
>
>Rabbitmq logs show this error:
>
>=ERROR REPORT==== 18-Jul-2014::14:06:21 ===
>Mnesia(rabbit@nodec22): ** ERROR ** mnesia_event got
>{inconsistent_database, starting_partitioned_network, rabbit@nodeh5}
>
>
>We will have to recluster rabbitmq.
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1344233
>
>Title:
> schema in a continuous loop and crashing
>
>Status in Juniper Openstack distribution:
> New
>
>Bug description:
> Build 1.10 2241
> env.roledefs = {
> 'all': [host1, host2, host3, host4, host5],
> 'cfgm': [host1,host4,host5],
> 'openstack': [host2],
> 'control': [host1,host4],
> 'compute': [host3],
> 'collector': [host1,host4,host5],
> 'webui': [host1], 'database': [host1,host4,host5],
> 'build': [host_build],
> }
>
> env.hostnames = {
> 'all': ['nodec22', 'nodeg30', 'nodeg29', 'nodeh4', 'nodeh5']
> }
>
> On this setup, i had a few test scripts which ran in parallel.
> There were 7-8 Vms which i deleted. After that schema on nodec22 is
>stuck some loop adding ACL, SG entries and it is repeatedly crashing as
>well.
>
> Logs of nodec22 will be in http://10.204.216.50/Docs/bugs/#
>
> -----
> schema.err:
>
> <type 'exceptions.AttributeError'>
> Python 2.7.3: /usr/bin/python
> Fri Jul 18 23:52:23 2014
>
> A problem occurred in a Python script. Here is the sequence of
> function calls leading up to the error, in the order they occurred.
>
> /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer
>object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
> 3366 result = arc_mapc.call('poll', pollreq)
> 3367 try:
> 3368 transformer.process_poll_result(result)
> 3369 except Exception as e:
> 3370 cgitb.Hook(
> transformer = <schema_transformer.to_bgp.SchemaTransformer object>
> transformer.process_poll_result = <bound method
>SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransfor
>mer object>>
> result = '<?xml version="1.0" encoding="UTF-8"
>standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'
>
> /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer
>object>, poll_result_str='<?xml version="1.0" encoding="UTF-8"
>standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
> 3023 pass
> 3024 else:
> 3025 func(idents, meta)
> 3026 # end for meta
> 3027 # end for result_type
> func = <bound method
>SchemaTransformer.add_virtual_netw...ema_transformer.to_bgp.SchemaTransfor
>mer object>>
> idents = {'network-ipam':
>'default-domain:default-project:default-network-ipam', 'virtual-network':
>'default-domain:TestBasicVMVN2-56418410:vn2228'}
> meta = <Element
>{http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-ipam>
>
> /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>add_virtual_network_network_ipam(self=<schema_transformer.to_bgp.SchemaTra
>nsformer object>, idents={'network-ipam':
>'default-domain:default-project:default-network-ipam', 'virtual-network':
>'default-domain:TestBasicVMVN2-56418410:vn2228'}, meta=<Element
>{http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-ipam>)
> 2706 subnet = VnSubnetsType()
> 2707 subnet.build(meta)
> 2708 virtual_network.ipams[ipam_name] = subnet
> 2709 virtual_network.update_ipams()
> 2710 # end add_virtual_network_network_ipam
> virtual_network = None
> virtual_network.ipams undefined
> ipam_name = 'default-domain:default-project:default-network-ipam'
> subnet = <vnc_api.gen.resource_xsd.VnSubnetsType object>
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
>'ipams'
> __class__ = <type 'exceptions.AttributeError'>
> __delattr__ = <method-wrapper '__delattr__' of
>exceptions.AttributeError object>
> __dict__ = {}
> __doc__ = 'Attribute not found.'
> __format__ = <built-in method __format__ of
>exceptions.AttributeError object>
> __getattribute__ = <method-wrapper '__getattribute__' of
>exceptions.AttributeError object>
> __getitem__ = <method-wrapper '__getitem__' of
>exceptions.AttributeError object>
> __getslice__ = <method-wrapper '__getslice__' of
>exceptions.AttributeError object>
> __hash__ = <method-wrapper '__hash__' of exceptions.AttributeError
>object>
> __init__ = <method-wrapper '__init__' of exceptions.AttributeError
>object>
> __new__ = <built-in method __new__ of type object>
> __reduce__ = <built-in method __reduce__ of
>exceptions.AttributeError object>
> __reduce_ex__ = <built-in method __reduce_ex__ of
>exceptions.AttributeError object>
> __repr__ = <method-wrapper '__repr__' of exceptions.AttributeError
>object>
> __setattr__ = <method-wrapper '__setattr__' of
>exceptions.AttributeError object>
> __setstate__ = <built-in method __setstate__ of
>exceptions.AttributeError object>
> __sizeof__ = <built-in method __sizeof__ of
>exceptions.AttributeError object>
> __str__ = <method-wrapper '__str__' of exceptions.AttributeError
>object>
> __subclasshook__ = <built-in method __subclasshook__ of type object>
> __unicode__ = <built-in method __unicode__ of
>exceptions.AttributeError object>
> args = ("'NoneType' object has no attribute 'ipams'",)
> message = "'NoneType' object has no attribute 'ipams'"
>
> The above is a description of an error in a Python program. Here is
> the original traceback:
>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py",
>line 3368, in launch_arc
> transformer.process_poll_result(result)
> File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py",
>line 3025, in process_poll_result
> func(idents, meta)
> File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py",
>line 2708, in add_virtual_network_network_ipam
> virtual_network.ipams[ipam_name] = subnet
> AttributeError: 'NoneType' object has no attribute 'ipams'
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/juniperopenstack/+bug/1344233/+subscriptions