schema_transformer crashes with AttributeError: 'NoneType' object has no attribute 'expand_connections'

[sajuptpm]

This happense because VirtualNetworkST.get(network_name) returns None.
Don't know why it returns none, any clue ?

Traceback
==========
<type 'exceptions.AttributeError'>
Python 2.7.6: /usr/bin/python
Mon Jun 22 01:13:23 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
 3651 result = arc_mapc.call('poll', pollreq)
 3652 try:
 3653 transformer.process_poll_result(result)
 3654 except Exception as e:
 3655 try:
transformer = <schema_transformer.to_bgp.SchemaTransformer object>
transformer.process_poll_result = <bound method SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransformer object>>
result = '<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'

 /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer object>, poll_result_str='<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
 3301 virtual_network = VirtualNetworkST.get(network_name)
 3302 old_virtual_network_connections =\
 3303 virtual_network.expand_connections()
 3304 old_service_chains = virtual_network.service_chains
 3305 virtual_network.connections = set()
virtual_network = None
virtual_network.expand_connections undefined
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'expand_connections'
    __class__ = <type 'exceptions.AttributeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.AttributeError object>
    __dict__ = {}
    __doc__ = 'Attribute not found.'
    __format__ = <built-in method __format__ of exceptions.AttributeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.AttributeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.AttributeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.AttributeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.AttributeError object>
    __init__ = <method-wrapper '__init__' of exceptions.AttributeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.AttributeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.AttributeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.AttributeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.AttributeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.AttributeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.AttributeError object>
    __str__ = <method-wrapper '__str__' of exceptions.AttributeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.AttributeError object>
    args = ("'NoneType' object has no attribute 'expand_connections'",)
    message = "'NoneType' object has no attribute 'expand_connections'"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3653, in launch_arc
    transformer.process_poll_result(result)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3303, in process_poll_result
    virtual_network.expand_connections()
AttributeError: 'NoneType' object has no attribute 'expand_connections'

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12159
Submitter: sajuptpm (<email address hidden>)

[Sachin Bansal (sbansal)]

As I mentioned in the review, it is not sufficient to just add the check for None. We need to find out why the network wasn't found. Do you know the steps to recreate the issue? If so, please enable debug traces in contrail-schema.conf and restart it before running the steps. We might get some clue on why it was not found.

[sajuptpm (sajuptpm)]

Hi sachin,
My findings:
Suppose we have 1000 virtual networks. when schema restart, it adds this 1000 virtual networks into current_network_set.
Then iterate through current_network_set and construct ACL entries and connectivity table, right?
During this current_network_set iteration, if user delete a virtual network, schema fails to query from Cassandra and end-up with this error.

[Sachin Bansal (sbansal)]

> On Jul 8, 2015, at 4:51 AM, sajuptpm <email address hidden> wrote:
>
> Hi sachin,
> My findings:
> Suppose we have 1000 virtual networks. when schema restart, it adds this 1000 virtual networks into current_network_set.
> Then iterate through current_network_set and construct ACL entries and connectivity table, right?
> During this current_network_set iteration, if user delete a virtual network, schema fails to query from Cassandra and end-up with this error.

This should not lead to this crash since any network for which delete hasn’t been received by schema transformer should still be present in VirtualNetworkST and get should succeed. And if schema transformer has already processed delete for this network, it will be removed from current_network_set.

Sachin

>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1470984
>
> Title:
> schema_transformer crashes with AttributeError: 'NoneType' object has
> no attribute 'expand_connections'
>
> Status in Juniper Openstack distribution:
> In Progress
> Status in Juniper Openstack trunk series:
> In Progress
> Status in OpenContrail:
> In Progress
>
> Bug description:
> This happense because VirtualNetworkST.get(network_name) returns None.
> Don't know why it returns none, any clue ?
>
> Traceback
> ==========
> <type 'exceptions.AttributeError'>
> Python 2.7.6: /usr/bin/python
> Mon Jun 22 01:13:23 2015
>
> A problem occurred in a Python script. Here is the sequence of
> function calls leading up to the error, in the order they occurred.
>
> /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in launch_arc(transformer=<schema_transformer.to_bgp.SchemaTransformer object>, ssrc_mapc=<cfgm_common.ifmap.client.client instance>)
> 3651 result = arc_mapc.call('poll', pollreq)
> 3652 try:
> 3653 transformer.process_poll_result(result)
> 3654 except Exception as e:
> 3655 try:
> transformer = <schema_transformer.to_bgp.SchemaTransformer object>
> transformer.process_poll_result = <bound method SchemaTransformer.process_poll_res...ema_transformer.to_bgp.SchemaTransformer object>>
> result = '<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>'
>
> /usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py in process_poll_result(self=<schema_transformer.to_bgp.SchemaTransformer object>, poll_result_str='<?xml version="1.0" encoding="UTF-8" standalone=...lResult></ns2:response></ns3:Body></ns3:Envelope>')
> 3301 virtual_network = VirtualNetworkST.get(network_name)
> 3302 old_virtual_network_connections =\
> 3303 virtual_network.expand_connections()
> 3304 old_service_chains = virtual_network.service_chains
> 3305 virtual_network.connections = set()
> virtual_network = None
> virtual_network.expand_connections undefined
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'expand_connections'
> __class__ = <type 'except...

[sajuptpm (sajuptpm)]

I have seen this issue in contrail R1.10.
Let me try to reproduce it.