Juniper Openstack

schema in a continuous loop and crashing

Bug #1344233 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
Critical
Sachin Bansal
Juniper Openstack r1.10-beta

Bug Description

Build 1.10 2241
env.roledefs = {
    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1,host4,host5],
    'openstack': [host2],
    'control': [host1,host4],
    'compute': [host3],
    'collector': [host1,host4,host5],
    'webui': [host1], 'database': [host1,host4,host5],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec22', 'nodeg30', 'nodeg29', 'nodeh4', 'nodeh5']
}

On this setup, i had a few test scripts which ran in parallel.
There were 7-8 Vms which i deleted. After that schema on nodec22 is stuck some loop adding ACL, SG entries and it is repeatedly crashing as well.

Logs of nodec22 will be in http://10.204.216.50/Docs/bugs/#

-----
schema.err:

<type 'exceptions.AttributeError'>
Python 2.7.3: /usr/bin/python
Fri Jul 18 23:52:23 2014

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
 3366 result = arc_mapc.call('poll', pollreq)
 3367 try:
 3368 transformer.process_poll_result(result)
 3369 except Exception as e:
 3370 cgitb.Hook(
transformer = <schema_transformer.to_bgp.SchemaTransformer object>
transformer.process_poll_result = <bound method SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransformer object>>
result = '<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer object>, poll_result_str='<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
 3023 pass
 3024 else:
 3025 func(idents, meta)
 3026 # end for meta
 3027 # end for result_type
func = <bound method SchemaTransformer.add_virtual_netw...ema_transformer.to_bgp.SchemaTransformer object>>
idents = {'network-ipam': 'default-domain:default-project:default-network-ipam', 'virtual-network': 'default-domain:TestBasicVMVN2-56418410:vn2228'}
meta = <Element {http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-ipam>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in add_virtual_network_network_ipam(self=<schema_transformer.to_bgp.SchemaTransformer object>, idents={'network-ipam': 'default-domain:default-project:default-network-ipam', 'virtual-network': 'default-domain:TestBasicVMVN2-56418410:vn2228'}, meta=<Element {http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-ipam>)
 2706 subnet = VnSubnetsType()
 2707 subnet.build(meta)
 2708 virtual_network.ipams[ipam_name] = subnet
 2709 virtual_network.update_ipams()
 2710 # end add_virtual_network_network_ipam
virtual_network = None
virtual_network.ipams undefined
ipam_name = 'default-domain:default-project:default-network-ipam'
subnet = <vnc_api.gen.resource_xsd.VnSubnetsType object>
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'ipams'
    __class__ = <type 'exceptions.AttributeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.AttributeError object>
    __dict__ = {}
    __doc__ = 'Attribute not found.'
    __format__ = <built-in method __format__ of exceptions.AttributeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.AttributeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.AttributeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.AttributeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.AttributeError object>
    __init__ = <method-wrapper '__init__' of exceptions.AttributeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.AttributeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.AttributeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.AttributeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.AttributeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.AttributeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.AttributeError object>
    __str__ = <method-wrapper '__str__' of exceptions.AttributeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.AttributeError object>
    args = ("'NoneType' object has no attribute 'ipams'",)
    message = "'NoneType' object has no attribute 'ipams'"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3368, in launch_arc
    transformer.process_poll_result(result)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3025, in process_poll_result
    func(idents, meta)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 2708, in add_virtual_network_network_ipam
    virtual_network.ipams[ipam_name] = subnet
AttributeError: 'NoneType' object has no attribute 'ipams'

Tags: config
Vedamurthy Joshi (vedujoshi)
summary: - schema in a continuous log and crashing
+ schema in a continuous loop and crashing
Revision history for this message
Sachin Bansal (sbansal) wrote :

I see the following in the logs:

07/19/2014 04:57:01 AM [nodec22:Schema:Config:0]: Exception Unknown id: Error: oper 1 url /fqname-to-id body {"fq_name": ["default-domain", "TestBasicVMVN4-45770645", "vn222"], "type": "virtual-network"} response Name [u'default-domain', u'TestBasicVMVN4-45770645', u'vn222'] not found while creating VirtualNetworkST for default-domain:TestBasicVMVN4-45770645:vn222

This means that schema transformer received this message from ifmap, but it wasn't there when it queried api server. On further debugging, I found that API server is not draining rabbit queues:

root@nodec22:~# rabbitmqctl list_queues
Listing queues ...
vnc_config.nodeh4-9100 950893
vnc_config.nodeh5-9100 948882
vnc_config.nodec22-9100 928005

Rabbitmq logs show this error:

=ERROR REPORT==== 18-Jul-2014::14:06:21 ===
Mnesia(rabbit@nodec22): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, rabbit@nodeh5}

We will have to recluster rabbitmq.

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote : Re: [Bug 1344233] Re: schema in a continuous loop and crashing
Download full text (7.6 KiB)

Sachin,
 Atleast from the command, the rabbit cluster seems to be fine ?
Ajay, Prakash,
 Any thoughts ?

root@nodec22:~/github/mine4/contrail-test# rabbitmqctl cluster_status
Cluster status of node rabbit@nodec22 ...
[{nodes,[{disc,[rabbit@nodeh5,rabbit@nodeh4,rabbit@nodec22]}]},
 {running_nodes,[rabbit@nodeh4,rabbit@nodeh5,rabbit@nodec22]}]
...done.
root@nodec22:~/github/mine4/contrail-test#

On 7/19/14, 5:05 AM, "Sachin Bansal" <email address hidden> wrote:

>I see the following in the logs:
>
>07/19/2014 04:57:01 AM [nodec22:Schema:Config:0]: Exception Unknown id:
>Error: oper 1 url /fqname-to-id body {"fq_name": ["default-domain",
>"TestBasicVMVN4-45770645", "vn222"], "type": "virtual-network"} response
>Name [u'default-domain', u'TestBasicVMVN4-45770645', u'vn222'] not found
>while creating VirtualNetworkST for default-
>domain:TestBasicVMVN4-45770645:vn222
>
>
>This means that schema transformer received this message from ifmap, but
>it wasn't there when it queried api server. On further debugging, I found
>that API server is not draining rabbit queues:
>
>root@nodec22:~# rabbitmqctl list_queues
>Listing queues ...
>vnc_config.nodeh4-9100 950893
>vnc_config.nodeh5-9100 948882
>vnc_config.nodec22-9100 928005
>
>
>Rabbitmq logs show this error:
>
>=ERROR REPORT==== 18-Jul-2014::14:06:21 ===
>Mnesia(rabbit@nodec22): ** ERROR ** mnesia_event got
>{inconsistent_database, starting_partitioned_network, rabbit@nodeh5}
>
>
>We will have to recluster rabbitmq.
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1344233
>
>Title:
> schema in a continuous loop and crashing
>
>Status in Juniper Openstack distribution:
> New
>
>Bug description:
> Build 1.10 2241
> env.roledefs = {
> 'all': [host1, host2, host3, host4, host5],
> 'cfgm': [host1,host4,host5],
> 'openstack': [host2],
> 'control': [host1,host4],
> 'compute': [host3],
> 'collector': [host1,host4,host5],
> 'webui': [host1], 'database': [host1,host4,host5],
> 'build': [host_build],
> }
>
> env.hostnames = {
> 'all': ['nodec22', 'nodeg30', 'nodeg29', 'nodeh4', 'nodeh5']
> }
>
> On this setup, i had a few test scripts which ran in parallel.
> There were 7-8 Vms which i deleted. After that schema on nodec22 is
>stuck some loop adding ACL, SG entries and it is repeatedly crashing as
>well.
>
> Logs of nodec22 will be in http://10.204.216.50/Docs/bugs/#
>
> -----
> schema.err:
>
> <type 'exceptions.AttributeError'>
> Python 2.7.3: /usr/bin/python
> Fri Jul 18 23:52:23 2014
>
> A problem occurred in a Python script. Here is the sequence of
> function calls leading up to the error, in the order they occurred.
>
> /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in
>launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer
>object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
> 3366 result = arc_mapc.call('poll', pollreq)
> 3367 try:
> 3368 transformer.process_poll_result(result)
> 3369 except Exception as e:
> 3370 ...

Read more...

Revision history for this message
alok kumar (kalok) wrote :
Download full text (4.7 KiB)

Build 1.10 2252:
hitting similar issue with below traceback:

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
 3371 result = arc_mapc.call('poll', pollreq)
 3372 try:
 3373 transformer.process_poll_result(result)
 3374 except Exception as e:
 3375 cgitb.Hook(
transformer = <schema_transformer.to_bgp.SchemaTransformer object>
transformer.process_poll_result = <bound method SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransformer object>>
result = '<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer object>, poll_result_str='<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
 3028 pass
 3029 else:
 3030 func(idents, meta)
 3031 # end for meta
 3032 # end for result_type
func = <bound method SchemaTransformer.add_virtual_netw...ema_transformer.to_bgp.SchemaTransformer object>>
idents = {'network-policy': 'default-domain:admin:policy0', 'virtual-network': 'default-domain:admin:vnet3'}
meta = <Element {http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-policy>

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in add_virtual_network_network_policy(self=<schema_transformer.to_bgp.SchemaTransformer object>, idents={'network-policy': 'default-domain:admin:policy0', 'virtual-network': 'default-domain:admin:vnet3'}, meta=<Element {http://www.contrailsystems.com/vnc_cfg.xsd}virtual-network-network-policy>)
 2697 policy = NetworkPolicyST.locate(policy_name)
 2698
 2699 policy.networks_back_ref.add(virtual_network.name)
 2700 vnp = VirtualNetworkPolicyType()
 2701 vnp.build(meta)
policy = <schema_transformer.to_bgp.NetworkPolicyST object>
policy.networks_back_ref = set([])
policy.networks_back_ref.add = <built-in method add of set object>
virtual_network = None
virtual_network.name undefined
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'name'
    __class__ = <type 'exceptions.AttributeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.AttributeError object>
    __dict__ = {}
    __doc__ = 'Attribute not found.'
    __format__ = <built-in method __format__ of exceptions.AttributeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.AttributeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.AttributeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.AttributeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.AttributeError object>
    __init__ = <method-wrapper '__init__' of exceptions.AttributeErr...

Read more...

Ganesha HV (ganeshahv)
Changed in juniperopenstack:
importance: High → Critical
Revision history for this message
Sachin Bansal (sbansal) wrote :

We will avoid crashing schema transformer by handling error in reading the network. The underlying rabbitmq issues will be fixed with a separate bug. https://review.opencontrail.org/#/c/1195/

Changed in juniperopenstack:
status: New → In Progress
Revision history for this message
Sachin Bansal (sbansal) wrote :

https://github.com/Juniper/contrail-controller/commit/b006f8dc8308e208c7df400fec75270ae4bf9438

Changed in juniperopenstack:
status: In Progress → Fix Committed
Vedamurthy Joshi (vedujoshi)
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.