DM: Config group delete messages seen. Entire DM config gets removed on mx

Bug #1702412 reported by Shashikiran H
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Released
High
Shashikiran H
Trunk
Fix Released
High
Shashikiran H

Bug Description

Version: 4.0.0.0-20 mitaka

I see DM intermittently sending the below delete message to mx. As a result, entire DM config vanishes on mx.

07/05/2017 06:25:40 AM [contrail-device-manager]:
send netconf message: <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos">
    <configuration>
        <groups operation="delete">
            <name>__contrail__</name>
        </groups>
        <apply-groups operation="delete">
            <name>__contrail__</name>
        </apply-groups>
    </configuration>
</config>

The testcase is just extending the VN to mx using DM. In DM log, I see multiple instances where this delete group message is being pushed to mx, so this is easily reproducible.

Revision history for this message
Suresh Balineni (sbalineni) wrote :

we have an UI bug - https://bugs.launchpad.net/juniperopenstack/+bug/1700674.
this will address the problem.

Please let me know if you are facing similar problem when you execute test scripts (but not from UI).

Revision history for this message
Shashikiran H (skiranh) wrote :

I will keep the setup in buggy state today. Please take a look.

The issue exists even without using UI. Just creating a physical device and attaching it to the contrail mx node deletes the __contrail__ group after about 10 min.

The entire multi node setup is in root@nodec22:/root/testbed.py
c22 has active device manager service.

Revision history for this message
Suresh Balineni (sbalineni) wrote :
Download full text (3.9 KiB)

Shashi,

I still see UI Authentication issue in this setup,

07/25/2017 04:19:59 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj deleted = True ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True >>
07/25/2017 04:20:22 PM [contrail-device-manager]: Router 10.204.217.190:
Peer (10.204.217.11) may not be configured multiple times in the same instance

07/25/2017 04:20:22 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:20:01 last_commit_duration = 21.8199260235 commit_status_message = failed to apply config, router response:
Peer (10.204.217.11) may not be configured multiple times in the same instance
  total_commits_sent_since_up = 4 >>

07/25/2017 04:22:58 PM [contrail-device-manager]: Router 10.204.217.190:
07/25/2017 04:22:58 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:22:23 last_commit_duration = 35.6202409267 commit_status_message = failed to apply config, router response: total_commits_sent_since_up = 1 >>
07/25/2017 04:22:59 PM [contrail-device-manager]: Router 10.204.217.190:
07/25/2017 04:23:39 PM [contrail-device-manager]: Router 10.204.217.190:
07/25/2017 04:23:39 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:23:06 last_commit_duration = 33.198638916 commit_status_message = failed to apply config, router response: total_commits_sent_since_up = 2 >>
07/25/2017 04:24:35 PM [contrail-device-manager]: Router 10.204.217.190: AuthenticationException('Authentication failed.',)
07/25/2017 04:25:08 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 connected_bgp_router = f7f31f15-926d-43d4-b7b5-486617b3a0d8 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:25:08 last_commit_duration = 24.4375770092 commit_status_message = success total_commits_sent_since_up = 3 >>
07/25/2017 04:27:26 PM [contrail-device-manager]: UvePhysicalRouterConfigTrace: data = << name = yuvaraj ip_address = 10.204.217.190 product_info = juniper:mx auto_conf_enabled = True netconf_enabled_status = True last_commit_time = 2017-07-25 16:27:26 last_commit_duration = 22.2161149979 commit_status_message = success total_commits_sent_since_up = 5 >>
07/25/2017 04:34:35 PM [contrail-...

Read more...

Revision history for this message
Suresh Balineni (sbalineni) wrote :

Noticed an Update BGpRouters Request from UI when this issue was happened,

Following is from contrail-web-ui Log (nodec22):

orchModels as: [ 'openstack' ]
07/25/2017 04:16:17 PM - ^[[34mdebug^[[39m: updateBGPRouter: /bgp-router/f7f31f15-926d-43d4-b7b5-486617b3a0d8
Didnt find the handler
Didnt find the handler

It looks like some updates are triggered from WebUI.

Revision history for this message
Suresh Balineni (sbalineni) wrote :

And this box does not have UI fix:

https://review.opencontrail.org/#/c/33406/1/webroot/config/physicaldevices/physicalrouters/api/physicalroutersconfig.api.js

https://review.opencontrail.org/#/c/33406/

Check the code: vi ./usr/src/contrail/contrail-web-controller/webroot/config/physicaldevices/physicalrouters/api/physicalroutersconfig.api.js Line Number 458

Revision history for this message
Shashikiran H (skiranh) wrote :

I am seeing this issue with R4.0 25th build.

Revision history for this message
Shashikiran H (skiranh) wrote :

I am NOT seeing this issue with R4.0 25 build.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.