schema in a continuous loop and crashing
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Juniper Openstack |
Fix Committed
|
Critical
|
Sachin Bansal |
Bug Description
Build 1.10 2241
env.roledefs = {
'all': [host1, host2, host3, host4, host5],
'cfgm': [host1,
'openstack': [host2],
'control': [host1,host4],
'compute': [host3],
'collector': [host1,
'webui': [host1], 'database': [host1,
'build': [host_build],
}
env.hostnames = {
'all': ['nodec22', 'nodeg30', 'nodeg29', 'nodeh4', 'nodeh5']
}
On this setup, i had a few test scripts which ran in parallel.
There were 7-8 Vms which i deleted. After that schema on nodec22 is stuck some loop adding ACL, SG entries and it is repeatedly crashing as well.
Logs of nodec22 will be in http://
-----
schema.err:
<type 'exceptions.
Python 2.7.3: /usr/bin/python
Fri Jul 18 23:52:23 2014
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/usr/lib/
3366 result = arc_mapc.
3367 try:
3368 transformer.
3369 except Exception as e:
3370 cgitb.Hook(
transformer = <schema_
transformer.
result = '<?xml version="1.0" encoding="UTF-8" standalone=
/usr/lib/
3023 pass
3024 else:
3025 func(idents, meta)
3026 # end for meta
3027 # end for result_type
func = <bound method SchemaTransform
idents = {'network-ipam': 'default-
meta = <Element {http://
/usr/lib/
2706 subnet = VnSubnetsType()
2707 subnet.build(meta)
2708 virtual_
2709 virtual_
2710 # end add_virtual_
virtual_network = None
virtual_
ipam_name = 'default-
subnet = <vnc_api.
<type 'exceptions.
__class__ = <type 'exceptions.
__delattr__ = <method-wrapper '__delattr__' of exceptions.
__dict__ = {}
__doc__ = 'Attribute not found.'
__format__ = <built-in method __format__ of exceptions.
__getattrib
__getitem__ = <method-wrapper '__getitem__' of exceptions.
__getslice__ = <method-wrapper '__getslice__' of exceptions.
__hash__ = <method-wrapper '__hash__' of exceptions.
__init__ = <method-wrapper '__init__' of exceptions.
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of exceptions.
__reduce_ex__ = <built-in method __reduce_ex__ of exceptions.
__repr__ = <method-wrapper '__repr__' of exceptions.
__setattr__ = <method-wrapper '__setattr__' of exceptions.
__setstate__ = <built-in method __setstate__ of exceptions.
__sizeof__ = <built-in method __sizeof__ of exceptions.
__str__ = <method-wrapper '__str__' of exceptions.
__subclassh
__unicode__ = <built-in method __unicode__ of exceptions.
args = ("'NoneType' object has no attribute 'ipams'",)
message = "'NoneType' object has no attribute 'ipams'"
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/usr/lib/
transformer
File "/usr/lib/
func(idents, meta)
File "/usr/lib/
virtual_
AttributeError: 'NoneType' object has no attribute 'ipams'
summary: |
- schema in a continuous log and crashing + schema in a continuous loop and crashing |
Changed in juniperopenstack: | |
importance: | High → Critical |
information type: | Proprietary → Public |
I see the following in the logs:
07/19/2014 04:57:01 AM [nodec22: Schema: Config: 0]: Exception Unknown id: Error: oper 1 url /fqname-to-id body {"fq_name": ["default-domain", "TestBasicVMVN4 -45770645" , "vn222"], "type": "virtual-network"} response Name [u'default-domain', u'TestBasicVMVN 4-45770645' , u'vn222'] not found while creating VirtualNetworkST for default- domain: TestBasicVMVN4- 45770645: vn222
This means that schema transformer received this message from ifmap, but it wasn't there when it queried api server. On further debugging, I found that API server is not draining rabbit queues:
root@nodec22:~# rabbitmqctl list_queues nodeh4- 9100 950893 nodeh5- 9100 948882 nodec22- 9100 928005
Listing queues ...
vnc_config.
vnc_config.
vnc_config.
Rabbitmq logs show this error:
=ERROR REPORT==== 18-Jul- 2014::14: 06:21 === rabbit@ nodec22) : ** ERROR ** mnesia_event got {inconsistent_ database, starting_ partitioned_ network, rabbit@nodeh5}
Mnesia(
We will have to recluster rabbitmq.