Exception AssertionError: continuously logged in nodemgr log file

[Sundaresan Rajangam]

The following exception ignored message is continuously logged in the <xyz>-nodemgr-strerr.log
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

[OpenContrail Admin (ci-admin-f)]

bug update...

[Robert Rosiak (robert-rosiak)]

I'm hitting this error as well. contrail-analytics-nodemgr-stderr.log file is flooded with this error:
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

contrail-analytics 3.0.0.0-2725

HA setup with 3 nodes.

[Praveen (pb0342)]

Me too facing this issue. Can we get any details on why are we seeing this issue. Have below questions in my mind.
+ Do we need to provide any remedy for this.
+ If log file /var/log/contrail/contrail-analytics-nodemgr-stderr.log is flooded with this error, and as per the state of this bug "wont fix" we can ignore this error. Will it impact space?

Please help me to understand. Thanks in advance.

[Jun Wei Wang (wjw7869)]

I also meeting this error, but could not know why it is ?

[Jiang Lu (lujiang)]

Hi Hari,

Could we have a fix to filter these being ignored AssertionError error logs? Customer opened JTAC case to ask to filter this error log in contrail-*nodemgr-stderr.log.

JTAC local testbed Contrail controller node and compute nodes also see the error:
CONT202: 10.85.19.202 root/Juniper
COMP204: 10.85.19.204 root/Juniper

root@cont202:/var/log/contrail# ls -alF *nodemgr-stderr*
-rw-r--r-- 1 root root 37815238 Dec 27 09:58 contrail-analytics-nodemgr-stderr.log
-rw-r--r-- 1 root root 2095893 Dec 27 09:58 contrail-config-nodemgr-stderr.log
-rw-r--r-- 1 root root 52428835 Dec 25 20:58 contrail-config-nodemgr-stderr.log.1
-rw-r--r-- 1 root root 35972584 Dec 27 09:58 contrail-control-nodemgr-stderr.log
-rw-r--r-- 1 root root 22118098 Dec 27 09:58 contrail-database-nodemgr-stderr.log
-rw-r--r-- 1 root root 52428846 Dec 16 00:18 contrail-database-nodemgr-stderr.log.1
root@cont202:/var/log/contrail#
root@cont202:/var/log/contrail#
root@cont202:/var/log/contrail# grep -rn AssertionError contrail-config-nodemgr-stderr.log | less
2:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
3:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
4:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
5:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
6:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
7:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
8:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
9:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
11:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
12:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
13:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
14:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
15:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
16:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
17:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
18:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
20:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
21:Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
22:Exception AssertionError: Assertion...

[Amjad (askhan)]

Hi Jack, please advise on the plan to develop the fix. Thank you

[Jiang Lu (lujiang)]

LP1466483 fix planned in Contrail 5.0.

[Jack Jonnalagadda (jackjvs)]

Some info on reasons for error:
https://stackoverflow.com/questions/8774958/keyerror-in-module-threading-after-a-successful-py-test-run

[Jack Jonnalagadda (jackjvs)]

More info: http://blog.codepainters.com/2012/11/20/gevent-monkey-patching-versus-sniffer-nose/

[Jack Jonnalagadda (jackjvs)]

=============== traceback begin =======
File "/usr/bin/contrail-nodemgr", line 9, in <module>
    load_entry_point('nodemgr==0.1dev', 'console_scripts', 'contrail-nodemgr')()
File "/usr/lib/python2.7/dist-packages/nodemgr/main.py", line 239, in main
    for line in traceback.format_stack():
=============== traceback end =======

[Jack Jonnalagadda (jackjvs)]

On 5.0 the following commands do not show the above exception:
journalctl -u contrail-analytics-nodemgr.service
journalctl -u contrail-analytics-nodemgr.service -b
systemctl status contrail-analytics-nodemgr.service

[Jack Jonnalagadda (jackjvs)]

root@cont202:/etc/contrail# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0.3.3-22 22

[Jiang Lu (lujiang)]

Hi Jack,

So you are saying Contrail 5.0 do not see these continuous "Exception AssertionError" any more?

And hence no need any fix in Contrail 5.0, right?

Thanks, Jiang Lu

[Jiang Lu (lujiang)]

Hi Jack,

Can we have the fix plan for this LP1466483 here?

Thanks, Jiang Lu

[Jiang Lu (lujiang)]

Hi Sundar,

Now I see this LP being assigned to you. Could you have a rough fix plan for this LP1466483?

Thanks, Jiang Lu

[Jiang Lu (lujiang)]

Can we reassign this LP as previous engineer Sundar being on leave?

[Jiang Lu (lujiang)]

Hi Santosh,

Can we have a fix plan for this LP1466483?

Understand it is a low priority LP, but AT&T still need the fix in 3.2.x.

Thanks, Jiang Lu

[Himanshu (bhimanshu)]

https://bugs.launchpad.net/juniperopenstack/+bug/1466483

Corresponding jira:
https://contrail-jws.atlassian.net/browse/JCB-173997

Thanks,
- Himanshu B.