DM: Config group delete messages seen. Entire DM config gets removed on mx

[Shashikiran H]

Version: 4.0.0.0-20 mitaka

I see DM intermittently sending the below delete message to mx. As a result, entire DM config vanishes on mx.

07/05/2017 06:25:40 AM [contrail-device-manager]:
send netconf message: <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos">
    <configuration>
        <groups operation="delete">
            <name>__contrail__</name>
        </groups>
        <apply-groups operation="delete">
            <name>__contrail__</name>
        </apply-groups>
    </configuration>
</config>

The testcase is just extending the VN to mx using DM. In DM log, I see multiple instances where this delete group message is being pushed to mx, so this is easily reproducible.

[Suresh Balineni (sbalineni)]

we have an UI bug - https://bugs.launchpad.net/juniperopenstack/+bug/1700674.
this will address the problem.

Please let me know if you are facing similar problem when you execute test scripts (but not from UI).

[Shashikiran H (skiranh)]

I will keep the setup in buggy state today. Please take a look.

The issue exists even without using UI. Just creating a physical device and attaching it to the contrail mx node deletes the __contrail__ group after about 10 min.

The entire multi node setup is in root@nodec22:/root/testbed.py
c22 has active device manager service.

[Suresh Balineni (sbalineni)]

Shashi,

I still see UI Authentication issue in this setup,

07/25/2017 04:19:59 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj deleted = True ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True >>
07/25/2017 04:20:22 PM [contrail-device-manager]: Router 10.204.217.190:
Peer (10.204.217.11) may not be configured multiple times in the same instance

07/25/2017 04:20:22 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:20:01 last_commit_duration = 21.8199260235 commit_status_message = failed to apply config, router response:
Peer (10.204.217.11) may not be configured multiple times in the same instance
  total_commits_sent_since_up = 4 >>

07/25/2017 04:22:58 PM [contrail-device-manager]: Router 10.204.217.190:
07/25/2017 04:22:58 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:22:23 last_commit_duration = 35.6202409267 commit_status_message = failed to apply config, router response: total_commits_sent_since_up = 1 >>
07/25/2017 04:22:59 PM [contrail-device-manager]: Router 10.204.217.190:
07/25/2017 04:23:39 PM [contrail-device-manager]: Router 10.204.217.190:
07/25/2017 04:23:39 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:23:06 last_commit_duration = 33.198638916 commit_status_message = failed to apply config, router response: total_commits_sent_since_up = 2 >>
07/25/2017 04:24:35 PM [contrail-device-manager]: Router 10.204.217.190: AuthenticationException('Authentication failed.',)
07/25/2017 04:25:08 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:25:08 last_commit_duration = 24.4375770092 commit_status_message = success total_commits_sent_since_up = 3 >>
07/25/2017 04:27:26 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:27:26 last_commit_duration = 22.2161149979 commit_status_message = success total_commits_sent_since_up = 5 >>
07/25/2017 04:34:35 PM [contrail-...

[Suresh Balineni (sbalineni)]

Noticed an Update BGpRouters Request from UI when this issue was happened,

Following is from contrail-web-ui Log (nodec22):

orchModels as: [ 'openstack' ]
07/25/2017 04:16:17 PM - ^[[34mdebug^[[39m: updateBGPRouter: /bgp-router/f7f31f15-926d-43d4-b7b5-486617b3a0d8
Didnt find the handler
Didnt find the handler

It looks like some updates are triggered from WebUI.

[Suresh Balineni (sbalineni)]

And this box does not have UI fix:

https://review.opencontrail.org/#/c/33406/1/webroot/config/physicaldevices/physicalrouters/api/physicalroutersconfig.api.js

https://review.opencontrail.org/#/c/33406/

Check the code: vi ./usr/src/contrail/contrail-web-controller/webroot/config/physicaldevices/physicalrouters/api/physicalroutersconfig.api.js Line Number 458

[Shashikiran H (skiranh)]

I am seeing this issue with R4.0 25th build.

[Shashikiran H (skiranh)]

I am NOT seeing this issue with R4.0 25 build.