Device manager not working. "No module named devices.default" message reported

[Shashikiran H]

Version: 2.20-92 juno

Topo:
nodec15, nodec26(1 cluster) ------ mx

2 Config, control, compute between nodec15 and nodec26

I configure a physical router in Physical Devices in webui and link it to BGP router in Infrastructure.

The device manager logs throw this message whenever I do the above or delete an already present peer:

vi /var/log/contrail/contrail-device-manager.log:

08/31/2015 11:05:46 PM [nodec15:DeviceManager:Config:0]: Router 10.204.217.190: No module named devices.default

I will attach the entire contrail-device-manager.log and contrail-device-manager-stdout.log.

Device manager does not push any config on the mx and the /var/log/messages on the mx box does not have any new messages during the above procedure.
I tried setting the log level to "DEBUG" and "SYS_NOTICE", but the log message still is the same as above

[Shashikiran H (skiranh)]

Debugged this with Suresh. The ncclient packages all point to /usr/lib/python2.7
root@nodec22:/opt/contrail/contrail_install_repo# dpkg -c python-ncclient_0.4.1-1contrail1_all.deb
drwxr-xr-x root/root 0 2014-05-02 07:42 ./
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/share/
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/share/doc-base/
-rw-r--r-- root/root 715 2014-04-29 09:51 ./usr/share/doc-base/python-ncclient
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/share/doc/
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/share/doc/python-ncclient/
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/
drwxr-xr-x root/root 0 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/_static/
-rw-r--r-- root/root 3500 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/comment-bright.png
-rw-r--r-- root/root 199 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/minus.png
-rw-r--r-- root/root 199 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/plus.png
-rw-r--r-- root/root 673 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/ajax-loader.gif
-rw-r--r-- root/root 3932 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/_static/pygments.css
-rw-r--r-- root/root 363 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/up.png
-rw-r--r-- root/root 5182 2014-02-22 14:36 ./usr/share/doc/python-ncclient/html/_static/logo.png
-rw-r--r-- root/root 8463 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/_static/basic.css
-rw-r--r-- root/root 3578 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/comment-close.png
-rw-r--r-- root/root 4040 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/_static/default.css
-rw-r--r-- root/root 368 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/down-pressed.png
-rw-r--r-- root/root 363 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/down.png
-rw-r--r-- root/root 392 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/file.png
-rw-r--r-- root/root 3445 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/comment.png
-rw-r--r-- root/root 372 2011-10-09 14:28 ./usr/share/doc/python-ncclient/html/_static/up-pressed.png
-rw-r--r-- root/root 5302 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/py-modindex.html
-rw-r--r-- root/root 57554 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/operations.html
-rw-r--r-- root/root 3563 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/extending.html
-rw-r--r-- root/root 12020 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/searchindex.js
-rw-r--r-- root/root 12090 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/xml_.html
-rw-r--r-- root/root 3535 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/search.html
-rw-r--r-- root/root 1453 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/objects.inv
-rw-r--r-- root/root 25625 2014-05-02 07:42 ./usr/share/doc/python-ncclient/html/genindex.html
...

[Suresh Balineni (sbalineni)]

Contrail builds ncclient 0.4.1 deb package and corresponding install rule says that all files should be installed in /usr/lib/python2.7/dist-packages/ncclient.

In the test setup, after installing contrail, some tests scripts are trying to install latest ncclient 0.4.3 version(in an indirect manner) which is violating the rules set by .deb package (0.4.1). When this installation step detects there is a version of ncclient 0.4.1, it removes all files before proceeding with the installation and then installs all files in '/usr/local/lib/python2.7/dist-packages/ncclient'.

this is cause of the problem.

Next step:

Contrail packager should build the latest ncclient(0.4.3) version, so that ncclient upgrades won't delete existing references.

[Shashikiran H (skiranh)]

Steps to workaround this bug:

1. If you have any pip installs to be done, install them if necessary.
2. Remove directory /usr/lib/python2.7/dist-packages/ncclient/
3. Restart Device manager using "service contrail-device-manager restart" command.

Device manager should be up now.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14226
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14227
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14280
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14227
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14226
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14282
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14283
Submitter: Shashikiran H (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14282
Submitter: Shashikiran H (<email address hidden>)

[Suresh Balineni (sbalineni)]

Shashi,

Can you follow up the review and make sure fixes went in?

This review is pending for longtime.

Thanks,
Suresh

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14226
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/f3b982c3efc63ee0a51372caf9deb2e761a478e9
Submitter: Zuul
Branch: R2.20

commit f3b982c3efc63ee0a51372caf9deb2e761a478e9
Author: skiranh <email address hidden>
Date: Mon Oct 5 12:11:58 2015 +0530

Including code to restart DM. This is because of bug 1490860. The temporary solution to the bug is to restart DM after pip installation of junos-eznc.
Removing sleep. Including code in run_tests.sh to take care of all services to be up.

Change-Id: Ibe134b0bb31cef45d6743360ad15ce24f424cbe0
Closes-Bug: #1490860

[OpenContrail Admin (ci-admin-f)]

bug update...

[Nagabhushana R (bhushana)]

This is a problem only when we run contrail-test on one of the CFGM nodes. Shouldn't be a concern on production environments