Juniper Openstack

DPDK: Agent stuck in init state

Bug #1578023 reported by Vinod Nair
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.0.2.0
Trunk
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.1.0.0-fcs

Bug Description

In 3.0.2.0-34, vrouter agent is stuck in init state

== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing
contrail-vrouter-dpdk active
contrail-vrouter-nodemgr active

root@cs-scale-7:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/0 PMD: 0 (Speed 10000, Duplex 1)
            Type:Physical HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:XTcL3L2VpDpdkVof MTU:1514 Ref:19
            RX port packets:8420 errors:0
            RX queue packets:5666 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:8420 bytes:970927 errors:0
            TX packets:9011 bytes:1584576 errors:0
            TX port packets:9002 errors:26

vif0/1 PMD: 0
            Type:Host HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:XL3L2Dpdk MTU:1514 Ref:12
            RX port packets:9011 errors:0
            RX queue packets:8735 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:9011 bytes:1584576 errors:0
            TX packets:8420 bytes:970927 errors:0
            TX queue packets:5666 errors:0
            TX port packets:8420 errors:0

root@cs-scale-7:~# ifconfig
bond0 Link encap:Ethernet HWaddr 90:e2:ba:4c:63:f0
          inet6 addr: fe80::b447:dff:fe46:bf77/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:1418 (1.4 KB)

bond0.141 Link encap:Ethernet HWaddr 0e:7e:59:0b:3d:c0
          inet6 addr: fe80::c7e:59ff:fe0b:3dc0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

em1 Link encap:Ethernet HWaddr c4:54:44:44:d7:05
          inet addr:10.87.141.7 Bcast:10.87.159.255 Mask:255.255.224.0
          inet6 addr: fe80::c654:44ff:fe44:d705/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:624336 errors:0 dropped:0 overruns:0 frame:0
          TX packets:79973 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:75297560 (75.2 MB) TX bytes:57182681 (57.1 MB)
          Memory:dfb20000-dfb40000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:2127 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:360945 (360.9 KB) TX bytes:360945 (360.9 KB)

vhost0 Link encap:Ethernet HWaddr 90:e2:ba:4c:63:f0
          inet addr:13.1.0.7 Bcast:13.1.0.255 Mask:255.255.255.0
          inet6 addr: fe80::92e2:baff:fe4c:63f0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:8432 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9027 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:972368 (972.3 KB) TX bytes:1588257 (1.5 MB)

virbr0 Link encap:Ethernet HWaddr 52:54:00:f8:99:0c
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

root@cs-scale-7:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/0 PMD: 0 (Speed 10000, Duplex 1)
            Type:Physical HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:XTcL3L2VpDpdkVof MTU:1514 Ref:19
            RX port packets:8432 errors:0
            RX queue packets:5676 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:8432 bytes:972368 errors:0
            TX packets:9027 bytes:1588257 errors:0
            TX port packets:9018 errors:26

vif0/1 PMD: 0
            Type:Host HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:XL3L2Dpdk MTU:1514 Ref:12
            RX port packets:9027 errors:0
            RX queue packets:8751 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:9027 bytes:1588257 errors:0
            TX packets:8432 bytes:972368 errors:0
            TX queue packets:5676 errors:0
            TX port packets:8432 errors:0

root@cs-scale-7:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 90:e2:ba:4c:63:f0
          inet addr:13.1.0.7 Bcast:13.1.0.255 Mask:255.255.255.0
          inet6 addr: fe80::92e2:baff:fe4c:63f0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:8435 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9029 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:972556 (972.5 KB) TX bytes:1588389 (1.5 MB)

 PDT cs-scale-7 [Thread 140123601299200, Pid 3806]: SANDESH: Send FAILED: 1462324090980700 TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 13.1.0.7:54135::13.1.0.4:8086(-1) < Active session connection complete controller/src/io/tcp_session.cc 263
2016-05-03 Tue 18:08:10:980.963 PDT cs-scale-7 [Thread 140123601299200, Pid 3806]: SANDESH: Send FAILED: 1462324090980815 NodeStatusUVE: data= [ name = cs-scale-7 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:13.1.0.4 server_addrs= [ [ (*_iter6) = 13.1.0.4:5269, ] ] status = Down description = Established ], [ type = XMPP name = dns-server:13.1.0.4 server_addrs= [ [ (*_iter6) = 13.1.0.4:53, ] ] status = Up description = OpenSent ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 13.1.0.4:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Discovery name = Collector server_addrs= [ [ (*_iter6) = 13.1.0.4:5998, ] ] status = Up description = SubscribeResponse ], [ type = Discovery name = dns-server server_addrs= [ [ (*_iter6) = 13.1.0.4:5998, ] ] status = Up description = SubscribeResponse ], [ type = Discovery name = xmpp-server server_addrs= [ [ (*_iter6) = 13.1.0.4:5998, ] ] status = Up description = SubscribeResponse ], ] ] description = XMPP:control-node:13.1.0.4, Collector connection down ], ] ] ]

root@cs-scale-7:~# netstat -antp | grep 5998
tcp 0 0 13.1.0.7:50831 13.1.0.4:5998 TIME_WAIT -
tcp 0 0 13.1.0.7:50824 13.1.0.4:5998 TIME_WAIT -
tcp 1 0 13.1.0.7:50663 13.1.0.4:5998 CLOSE_WAIT 3146/python

 The cluster was upgraded from build 33 to 34 , even fresh install shows the same

See original description
Tags: blocker dpdk
Jeba Paulaiyan (jebap)
information type: Proprietary → Public
Vinod Nair (vinodnair)
description: updated
Revision history for this message
Raja Sivaramakrishnan (raja-u) wrote :

Agent is not creating pkt0 interface (with DPDK) and agent introspect is not accessible. Agent binary from build 32 works fine.
Can somebody take a look? Thanks.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19933
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19933
Committed: http://github.org/Juniper/contrail-controller/commit/3908057e1ab2fdc2fb11a23c477e852e3f4eee80
Submitter: Zuul
Branch: R3.0

commit 3908057e1ab2fdc2fb11a23c477e852e3f4eee80
Author: Hari <email address hidden>
Date: Thu May 5 20:09:33 2016 +0530

Agent init in DPDK is dependent on asio thread.

Do not block start of event manager.

Change-Id: I19b0e11afbb43d9f47d89a07e38cceb4a85f09b0
closes-bug: #1578023

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19954
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19954
Committed: http://github.org/Juniper/contrail-controller/commit/0bd89dbfd5ceb3d92d320eae078f7ad8950630c1
Submitter: Zuul
Branch: master

commit 0bd89dbfd5ceb3d92d320eae078f7ad8950630c1
Author: Hari <email address hidden>
Date: Thu May 5 20:09:33 2016 +0530

Agent init in DPDK is dependent on asio thread.

Do not block start of event manager.

Change-Id: I19b0e11afbb43d9f47d89a07e38cceb4a85f09b0
closes-bug: #1578023
(cherry picked from commit 3908057e1ab2fdc2fb11a23c477e852e3f4eee80)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.