contrail-device-manager crashing repeatedly in uve_send() in db.py

Bug #1466005 reported by Vedamurthy Joshi
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Suresh Balineni
Trunk
Fix Committed
High
Suresh Balineni

Bug Description

R2.20 Build 53 Ubuntu 14.04 Multi-node setup

It is seen that contrail-device-manager is crashing repeatedly with below trace :
This box was upgraded from an earlier 2.20 build to 2.20 Build53
Logs on this node will be uploaded to http://10.204.216.50/Docs/bugs/#

root@nodec1:/var/log/contrail# grep "A problem occurred in a Python script" contrail-device-manager-stdout.log | wc -l
3512 <<<<<<<
root@nodec1:/var/log/contrail#

<type 'exceptions.TypeError'>
Python 2.7.6: /usr/bin/python
Wed Jun 17 14:18:25 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/bin/contrail-device-manager in <module>()
    6
    7 if __name__ == '__main__':
    8 sys.exit(
    9 load_entry_point('device-manager==0.1dev', 'console_scripts', 'contrail-device-manager')()
   10 )
load_entry_point = <function load_entry_point>

 /usr/lib/python2.7/dist-packages/device_manager/device_manager.py in server_main()
  534 def server_main():
  535 cgitb.enable(format='text')
  536 main()
  537 # end server_main
  538
global main = <function main>

 /usr/lib/python2.7/dist-packages/device_manager/device_manager.py in main(args_str='--conf_file /etc/contrail/contrail-device-manage...nf_file /etc/contrail/contrail-keystone-auth.conf')
  523 _zookeeper_client.master_election(zk_path_pfx+"/device-manager",
  524 os.getpid(), run_device_manager,
  525 args)
  526 # end main
  527
args = Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181')

 /usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py in master_election(self=<cfgm_common.zkclient.ZookeeperClient object>, path='/device-manager', identifier=29489, func=<function run_device_manager>, *args=(Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'),), **kwargs={})
  289 while True:
  290 self._election = self._zk_client.Election(path, identifier)
  291 self._election.run(self._zk_election_callback, func, *args, **kwargs)
  292 # end master_election
  293
self = <cfgm_common.zkclient.ZookeeperClient object>
self._election = <kazoo.recipe.election.Election object>
self._election.run = <bound method Election.run of <kazoo.recipe.election.Election object>>
self._zk_election_callback = <bound method ZookeeperClient._zk_election_callb...of <cfgm_common.zkclient.ZookeeperClient object>>
func = <function run_device_manager>
args = (Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'),)
kwargs = {}

 /usr/lib/python2.7/dist-packages/kazoo/recipe/election.py in run(self=<kazoo.recipe.election.Election object>, func=<bound method ZookeeperClient._zk_election_callb...of <cfgm_common.zkclient.ZookeeperClient object>>, *args=(<function run_device_manager>, Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181')), **kwargs={})
   46 try:
   47 with self.lock:
   48 func(*args, **kwargs)
   49
   50 except CancelledError:
func = <bound method ZookeeperClient._zk_election_callb...of <cfgm_common.zkclient.ZookeeperClient object>>
args = (<function run_device_manager>, Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'))
kwargs = {}

 /usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py in _zk_election_callback(self=<cfgm_common.zkclient.ZookeeperClient object>, func=<function run_device_manager>, *args=(Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'),), **kwargs={})
 /usr/lib/python2.7/dist-packages/kazoo/recipe/election.py in run(self=<kazoo.recipe.election.Election object>, func=<bound method ZookeeperClient._zk_election_callb...of <cfgm_common.zkclient.ZookeeperClient object>>, *args=(<function run_device_manager>, Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181')), **kwargs={})
   46 try:
   47 with self.lock:
   48 func(*args, **kwargs)
   49
   50 except CancelledError:
func = <bound method ZookeeperClient._zk_election_callb...of <cfgm_common.zkclient.ZookeeperClient object>>
args = (<function run_device_manager>, Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'))
kwargs = {}

 /usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py in _zk_election_callback(self=<cfgm_common.zkclient.ZookeeperClient object>, func=<function run_device_manager>, *args=(Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'),), **kwargs={})
  281
  282 def _zk_election_callback(self, func, *args, **kwargs):
  283 func(*args, **kwargs)
  284 # Exit if running master encounters error or exception
  285 exit(1)
func = <function run_device_manager>
args = (Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'),)
kwargs = {}

 /usr/lib/python2.7/dist-packages/device_manager/device_manager.py in run_device_manager(args=Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'))
  528
  529 def run_device_manager(args):
  530 device_manager = DeviceManager(args)
  531 # end run_device_manager
  532
device_manager undefined
global DeviceManager = <class 'device_manager.device_manager.DeviceManager'>
args = Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181')

 /usr/lib/python2.7/dist-packages/device_manager/device_manager.py in __init__(self=<device_manager.device_manager.DeviceManager object>, args=Namespace(admin_password='contrail123', admin_te...:2181,10.204.216.60:2181', zk_server_port='2181'))
  223 else:
  224 for fq_name, uuid in pr_list:
  225 pr = PhysicalRouterDM.locate(uuid)
  226 if pr.bgp_router:
  227 BgpRouterDM.locate(pr.bgp_router)
pr undefined
global PhysicalRouterDM = <class 'device_manager.db.PhysicalRouterDM'>
PhysicalRouterDM.locate = <bound method __metaclass__.locate of <class 'device_manager.db.PhysicalRouterDM'>>
uuid = u'07689690-e248-40d7-8eec-3b9adcc079cd'

 /usr/lib/python2.7/dist-packages/cfgm_common/vnc_db.py in locate(cls=<class 'device_manager.db.PhysicalRouterDM'>, key=u'07689690-e248-40d7-8eec-3b9adcc079cd', *args=())
   53 if key not in cls._dict:
   54 try:
   55 cls._dict[key] = cls(key, *args)
   56 except NoIdError as e:
   57 cls._logger.debug(
cls = <class 'device_manager.db.PhysicalRouterDM'>
cls._dict = {}
key = u'07689690-e248-40d7-8eec-3b9adcc079cd'
args = ()

 /usr/lib/python2.7/dist-packages/device_manager/db.py in __init__(self=<device_manager.db.PhysicalRouterDM object>, uuid=u'07689690-e248-40d7-8eec-3b9adcc079cd', obj_dict=None)
  102 self.management_ip, self.user_credentials, self.vendor,
  103 self.product, self.vnc_managed, self._logger)
  104 self.uve_send()
  105 # end __init__
  106
self = <device_manager.db.PhysicalRouterDM object>
self.uve_send = <bound method PhysicalRouterDM.uve_send of <device_manager.db.PhysicalRouterDM object>>

 /usr/lib/python2.7/dist-packages/device_manager/db.py in uve_send(self=<device_manager.db.PhysicalRouterDM object>, deleted=False)
  290 connected_bgp_router=self.bgp_router,
  291 auto_conf_enabled=self.vnc_managed,
  292 product_info=self.vendor + ':' + self.product)
  293 if deleted:
  294 pr_trace.deleted = True
product_info undefined
self = <device_manager.db.PhysicalRouterDM object>
self.vendor = u'Juniper'
self.product = None
<type 'exceptions.TypeError'>: coercing to Unicode: need string or buffer, NoneType found
    __class__ = <type 'exceptions.TypeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.TypeError object>
    __dict__ = {}
    __doc__ = 'Inappropriate argument type.'
    __format__ = <built-in method __format__ of exceptions.TypeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.TypeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.TypeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.TypeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.TypeError object>
    __init__ = <method-wrapper '__init__' of exceptions.TypeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.TypeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.TypeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.TypeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.TypeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.TypeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.TypeError object>
    __str__ = <method-wrapper '__str__' of exceptions.TypeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.TypeError object>
    args = ('coercing to Unicode: need string or buffer, NoneType found',)
    message = 'coercing to Unicode: need string or buffer, NoneType found'

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/bin/contrail-device-manager", line 9, in <module>
    load_entry_point('device-manager==0.1dev', 'console_scripts', 'contrail-device-manager')()
  File "/usr/lib/python2.7/dist-packages/device_manager/device_manager.py", line 536, in server_main
    main()
  File "/usr/lib/python2.7/dist-packages/device_manager/device_manager.py", line 525, in main
    args)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py", line 291, in master_election
    self._election.run(self._zk_election_callback, func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kazoo/recipe/election.py", line 48, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py", line 283, in _zk_election_callback
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/device_manager/device_manager.py", line 530, in run_device_manager
    device_manager = DeviceManager(args)
  File "/usr/lib/python2.7/dist-packages/device_manager/device_manager.py", line 225, in __init__
    pr = PhysicalRouterDM.locate(uuid)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_db.py", line 55, in locate
    cls._dict[key] = cls(key, *args)
  File "/usr/lib/python2.7/dist-packages/device_manager/db.py", line 104, in __init__
    self.uve_send()
  File "/usr/lib/python2.7/dist-packages/device_manager/db.py", line 292, in uve_send
    product_info=self.vendor + ':' + self.product)
TypeError: coercing to Unicode: need string or buffer, NoneType found

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6, host7],
    'cfgm': [host1,host2,host3],
    'openstack': [host1,host2,host3],
    'control': [host1,host2,host3],
    'compute': [host4,host5, host6, host7],
    'collector': [host1,host2,host3],
    'webui': [host1],
    'database': [host1,host2,host3],
    'toragent': [host6, host7],
    'tsn': [host6, host7],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec1', 'nodec2', 'nodec3', 'nodek1', 'nodek2', 'nodek3', 'nodeg11']
}

Tags: config
Changed in juniperopenstack:
milestone: none → r2.30-fcs
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/11751
Submitter: Suresh Balineni (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/11751
Committed: http://github.org/Juniper/contrail-controller/commit/56233ca7c050b43c971135e15707bb557506bd02
Submitter: Zuul
Branch: R2.20

commit 56233ca7c050b43c971135e15707bb557506bd02
Author: sbalineni <email address hidden>
Date: Wed Jun 17 08:04:09 2015 -0700

DM: handle gracefully if product or vendor name is not specified

Closes-Bug: #1466005

Change-Id: I646fd33244c0239bde5bdf7b90bdf6958126e335

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/11801
Submitter: Suresh Balineni (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/11801
Committed: http://github.org/Juniper/contrail-controller/commit/5ce81ce45797a8eddb33fc37a8259d008a7e7edc
Submitter: Zuul
Branch: master

commit 5ce81ce45797a8eddb33fc37a8259d008a7e7edc
Author: sbalineni <email address hidden>
Date: Thu Jun 18 12:00:47 2015 -0700

DM: porting fixes from R2.20 to master

Closes-Bug: #1464448
Closes-Bug: #1464453
Closes-Bug: #1464456
Closes-Bug: #1465904
Closes-Bug: #1466005

Change-Id: Iad3077505507e798d70258d76b40380527410eeb

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.