contrail-vrouter-dpdk crash with signal SIGSEGV @dpdk_adjust_tcp_mss
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.0 |
New
|
High
|
Jeya ganesh babu J | |||
R3.1 |
New
|
High
|
Jeya ganesh babu J | |||
R3.2 |
New
|
High
|
Jeya ganesh babu J | |||
R4.0 |
New
|
High
|
Jeya ganesh babu J | |||
R4.1 |
New
|
High
|
Jeya ganesh babu J | |||
R5.0 |
New
|
High
|
Jeya ganesh babu J | |||
Trunk |
New
|
High
|
Jeya ganesh babu J |
Bug Description
Field reported contrail-
vrouter-dpdk and vrouter-agent core files now uploaded at 10.85.180.80 (comp80) /root/jlu/
root@bcomp80:
total 5324056
drwxr-xr-x 2 root root 4096 Apr 13 14:10 ./
drwx------ 7 root root 4096 Apr 13 13:40 ../
-rw-r--r-- 1 root root 479736974 Mar 4 10:20 contrail-
-rw-r--r-- 1 root root 17885993 Mar 4 10:58 contrail-
-rw------- 1 root root 1521876992 Apr 10 06:57 core.contrail-
-rw------- 1 root root 3432312832 Apr 10 06:56 core.lcore-
root@bcomp80:
...
Core was generated by `/usr/bin/
Program terminated with signal SIGSEGV, Segmentation fault.
#0 dpdk_adjust_tcp_mss (tcph=0x7f2938c
889 vrouter/
(gdb) bt
#0 dpdk_adjust_tcp_mss (tcph=0x7f2938c
#1 0x0000000000534371 in dpdk_pkt_
#2 0x000000000054f058 in vr_bridge_input (router=<optimized out>, pkt=0x7f2938ce8b40, fmd=0x7f4315e725b0) at vrouter/
#3 0x0000000000552ed5 in vr_flow_flush_pnode (fmd=0x7f4315e7
#4 __vr_flow_
#5 vr_flush_entry (flmd=<optimized out>, fmd=0x7f4315e725b0, fe=0x2aaacd737800, router=0x55e17c0 <router>) at vrouter/
#6 __vr_flow_work (flmd=0x7f292eb
#7 vr_flow_work (arg=0x7f292ebc
#8 0x000000000052f529 in dpdk_schedule_work (cpu=<optimized out>, fn=<optimized out>, arg=<optimized out>) at vrouter/
#9 0x0000000000553dd3 in vr_flow_
#10 vr_flow_set (router=<optimized out>, flow_resp=
#11 vr_flow_req_process (s_req=
#12 0x000000000059db94 in sandesh_decode_one (error=
#13 sandesh_decode (buf=0x7f293039a4d8 "", buf_len=262, sinfo_find_
#14 0x000000000056f7e8 in sandesh_
#15 0x0000000000561028 in vr_message_request (message=
#16 0x000000000053f55d in dpdk_netlink_
#17 0x000000000054487a in usock_read_done (usockp=
#18 vr_usocket_read (usockp=
#19 0x0000000000545b1d in vr_usocket_io (transport=
#20 0x000000000053eabd in dpdk_lcore_
#21 vr_dpdk_
#22 0x00000000004425f5 in eal_thread_loop (arg=<optimized out>) at /home/contrail-
#23 0x00007f4318450184 in start_thread () from /lib/x86_
#24 0x00007f4317f75ffd in qfcvt_r () from /lib/x86_
#25 0x0000000000000000 in ?? ()
root@bcomp80:
...
Core was generated by `/usr/bin/
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb726174c37 in ?? () from /lib/x86_
(gdb) bt
#0 0x00007fb726174c37 in ?? () from /lib/x86_
#1 0x0000000000002929 in ?? ()
#2 0x7328203c20746e63 in ?? ()
#3 0x5f2820666f657a69 in ?? ()
#4 0x65756c61765f6c6e in ?? ()
#5 0x434c5f657079745f in ?? ()
#6 0x524154454e4f4d5f in ?? ()
#7 0x7a6973202f202959 in ?? ()
#8 0x6c6e5f2820666f65 in ?? ()
#9 0x745f65756c61765f in ?? ()
mtn6r03c005:
Package Version Build-ID | Repo | Package Name
-------
contrail-lib 3.2.9.0-69 69
contrail-nodemgr 3.2.9.0-69 69
contrail-nova-vif 3.2.9.0-69 69
contrail-
contrail-setup 3.2.9.0-69 69
contrail-utils 3.2.9.0-69 69
contrail-
contrail-
contrail-
contrail-
contrail-
contrail-
nova-common 1:2015.
nova-compute 1:2015.
nova-compute-kvm 1:2015.
python-contrail 3.2.9.0-69 69
python-
python-
python-nova 1:2015.
python-
mtn6r03c005:
== Contrail vRouter ==
supervisor-vrouter: active
contrail-
contrail-
contrail-
========Run time service failures=
/var/crashes/
/var/crashes/
/var/crashes/
root@mtn6r03c005:~# cd /var/crashes/
root@mtn6r03c00
total 4947968
-rw------- 1 root root 1521876992 Apr 10 13:57 core.contrail-
-rw------- 1 root root 3432312832 Apr 10 13:56 core.lcore-
root@bcomp80:~/jlu# date -d @1523368617 <<<<
Tue Apr 10 06:56:57 PDT 2018
root@bcomp80:~/jlu# date -d @1523368630
Tue Apr 10 06:57:10 PDT 2018
/var/log/contrail/ uploaded at 10.85.180.80 (comp80) /root/jlu/
root@bcomp80:
total 544524
drwxr-x--- 2 contrail adm 4096 Apr 13 14:53 ./
drwxr-xr-x 3 root root 4096 Apr 13 14:26 ../
-rw-r--r-- 1 contrail adm 12257 Apr 9 13:27 contrail-
-rw-r--r-- 1 contrail adm 6158191 Apr 13 12:37 contrail-
-rw-r--r-- 1 contrail adm 23852991 Apr 13 12:53 contrail-
-rw-r--r-- 1 root root 50484623 Apr 13 13:34 contrail-
-rw-r--r-- 1 root root 52428814 Apr 5 08:23 contrail-
-rw-r--r-- 1 contrail contrail 52428814 Mar 26 21:39 contrail-
-rw-r--r-- 1 contrail adm 52428809 Mar 4 19:13 contrail-
-rw-r--r-- 1 contrail adm 52428843 Feb 9 17:22 contrail-
-rw-r--r-- 1 contrail adm 52428818 Jan 17 04:46 contrail-
-rw-r--r-- 1 contrail adm 52428804 Dec 25 04:35 contrail-
-rw-r--r-- 1 contrail adm 52428819 Dec 2 05:40 contrail-
-rw-r--r-- 1 contrail adm 52428852 Nov 8 07:27 contrail-
-rw-r--r-- 1 contrail adm 52428864 Oct 17 04:04 contrail-
-rw-r--r-- 1 contrail contrail 5085319 Apr 13 13:34 contrail-
-rw-r--r-- 1 contrail contrail 1556 Apr 13 12:37 process_
-rw-r--r-- 1 contrail adm 78677 Apr 13 12:37 supervisord-
supervisord-
2018-03-29 21:05:38,691 INFO success: contrail-
2018-04-10 13:57:09,672 INFO exited: contrail-
2018-04-10 13:57:10,022 INFO spawned: 'contrail-
2018-04-10 13:57:11,379 INFO success: contrail-
2018-04-10 13:57:18,768 INFO exited: contrail-
2018-04-10 13:57:19,771 INFO spawned: 'contrail-
2018-04-10 13:57:24,890 INFO success: contrail-
contrail-
04/10/2018 01:57:04 PM [contrail-
04/10/2018 01:57:04 PM [contrail-
04/10/2018 01:57:04 PM [contrail-
wokeup and found a line
contrail-
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
find command option for cores:find /var/crashes
-name core.[A-
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
core file: /var/crashes/
core_file_list:[], self.max_cores:4
# of cores for contrail-
04/10/2018 01:57:10 PM [contrail-
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
contrail is not in the sudoers file. This incident will be reported.
04/10/2018 01:57:10 PM [contrail-
wokeup and found a line
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
contrail is not in the sudoers file. This incident will be reported.
04/10/2018 01:57:10 PM [contrail-
wokeup and found a line
04/10/2018 01:57:12 PM [contrail-
wokeup and found a line
contrail-
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
find command option for cores:find /var/crashes
-name core.[A-
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/
core file: /var/crashes/
core_file_list:[], self.max_cores:4
# of cores for contrail-
04/10/2018 01:57:19 PM [contrail-
wokeup and found a line
wokeup and found a line
04/10/2018 01:57:25 PM [contrail-
contrail-
...
2018-04-10 13:47:30,872 UVHOST: Client _tap374a1fc8-f6: vring 1 is ready
2018-04-10 13:47:30,872 UVHOST: Client _tap374a1fc8-f6: no handler defined for message 12
2018-04-10 13:51:55,460 UVHOST: Client _tap35caf2ed-8a: handling message 13
2018-04-10 13:51:55,460 UVHOST: SET VRING CALL: vring 0 FD 249
2018-04-10 13:51:55,460 UVHOST: Client _tap35caf2ed-8a: handling message 13
2018-04-10 13:51:55,460 UVHOST: SET VRING CALL: vring 0 FD 250
2018-04-10 13:51:59,458 UVHOST: Client _tap374a1fc8-f6: handling message 13
2018-04-10 13:51:59,458 UVHOST: SET VRING CALL: vring 0 FD 249
2018-04-10 13:51:59,472 UVHOST: Client _tap374a1fc8-f6: handling message 13
2018-04-10 13:51:59,472 UVHOST: SET VRING CALL: vring 0 FD 251
2018-04-10 13:57:10,030 VROUTER: vRouter/DPDK version: {"build-info": [{"build-time": "2018-03-04 18:54:00.499936", "build-hostname": "ubuntu", "build-user": "contrail-builder", "build-version": "3.2.9.0"}]}
2018-04-10 13:57:10,082 VROUTER: Max HOLD flow entries set to 1000
2018-04-10 13:57:10,083 VROUTER: Using VLAN TCI: 2004
2018-04-10 13:57:10,083 VROUTER: Bridge Table limit: 262144
2018-04-10 13:57:10,083 VROUTER: Bridge Table overflow limit: 0
2018-04-10 13:57:10,083 VROUTER: Flow Table limit: 2097152
2018-04-10 13:57:10,083 VROUTER: Flow Table overflow limit: 419840
2018-04-10 13:57:10,083 VROUTER: MPLS labels limit: 6000
2018-04-10 13:57:10,083 VROUTER: Nexthops limit: 65536
2018-04-10 13:57:10,083 VROUTER: VRF tables limit: 4096
2018-04-10 13:57:10,083 VROUTER: Packet pool size: 16384
2018-04-10 13:57:10,083 VROUTER: Maximum packet size: 9216
2018-04-10 13:57:10,083 VROUTER: EAL arguments:
2018-04-10 13:57:10,083 VROUTER: -n "4"
2018-04-10 13:57:10,083 VROUTER: --socket-mem "1024,1024"
2018-04-10 13:57:10,083 VROUTER: --vdev "eth_bond_
2018-04-10 13:57:10,083 VROUTER: --lcores "(0-2)@
2018-04-10 13:57:10,083 EAL: Detected 56 lcore(s)
2018-04-10 13:57:10,098 EAL: Probing VFIO support...
2018-04-10 13:57:58,333 EAL: PCI device 0000:01:00.0 on NUMA socket 0
2018-04-10 13:57:58,333 EAL: probe driver: 8086:10fb net_ixgbe
2018-04-10 13:57:58,333 EAL: PCI device 0000:01:00.1 on NUMA socket 0
2018-04-10 13:57:58,333 EAL: probe driver: 8086:10fb net_ixgbe
2018-04-10 13:57:58,333 EAL: PCI device 0000:03:00.0 on NUMA socket 0
2018-04-10 13:57:58,333 EAL: probe driver: 8086:154d net_ixgbe
2018-04-10 13:57:58,472 EAL: PCI device 0000:03:00.1 on NUMA socket 0
2018-04-10 13:57:58,472 EAL: probe driver: 8086:154d net_ixgbe
2018-04-10 13:57:58,472 EAL: PCI device 0000:06:00.0 on NUMA socket 0
2018-04-10 13:57:58,472 EAL: probe driver: 8086:1521 net_e1000_igb
2018-04-10 13:57:58,472 EAL: PCI device 0000:06:00.1 on NUMA socket 0
2018-04-10 13:57:58,472 EAL: probe driver: 8086:1521 net_e1000_igb
2018-04-10 13:57:58,472 EAL: PCI device 0000:83:00.0 on NUMA socket 1
2018-04-10 13:57:58,472 EAL: probe driver: 8086:154d net_ixgbe
2018-04-10 13:57:58,472 EAL: PCI device 0000:83:00.1 on NUMA socket 1
2018-04-10 13:57:58,472 EAL: probe driver: 8086:154d net_ixgbe
2018-04-10 13:57:58,612 EAL: Initializing pmd_bond for eth_bond_bond0
2018-04-10 13:57:58,612 PMD: Using mode 4, it is necessary to do TX burst and RX burst at least every 100ms.
2018-04-10 13:57:58,612 EAL: Create bonded device eth_bond_bond0 on port 2 in mode 4 on socket 1.
2018-04-10 13:57:58,638 VROUTER: Found 3 eth device(s)
2018-04-10 13:57:58,638 VROUTER: Using 8 forwarding lcore(s)
2018-04-10 13:57:58,638 VROUTER: Using 0 IO lcore(s)
2018-04-10 13:57:58,638 VROUTER: Using 5 service lcores
2018-04-10 13:57:58,638 VROUTER: set fd limit to 4096 (prev 1024, max 4096)
2018-04-10 13:57:58,670 VROUTER: Adding VLAN forwarding interface bond0
2018-04-10 13:57:58,670 VROUTER: initializing KNI with 16 maximum interfaces
2018-04-10 13:57:58,671 VROUTER: bind KNI kernel thread to CPU 8
2018-04-10 13:57:58,672 KNI: pci: 03:00:00 8086:154d
2018-04-10 13:57:58,672 VROUTER: Starting NetLink...
2018-04-10 13:57:58,672 VROUTER: Lcore 14: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 VROUTER: Lcore 10: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 VROUTER: Lcore 13: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 USOCK: usock_alloc[
2018-04-10 13:57:58,672 USOCK: usock_alloc[
2018-04-10 13:57:58,672 VROUTER: Lcore 16: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 VROUTER: Lcore 11: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 VROUTER: Lcore 17: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 USOCK: usock_alloc[
Buffer size set to 18320000 (requested 9216000)
2018-04-10 13:57:58,672 VROUTER: Lcore 12: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 VROUTER: Lcore 15: distributing MPLSoGRE packets to [10,11,
2018-04-10 13:57:58,672 VROUTER: NetLink TCP socket FD is 62
2018-04-10 13:57:58,672 VROUTER: uvhost Unix socket FD is 63
2018-04-10 13:57:58,674 UVHOST: Starting uvhost server...
2018-04-10 13:57:58,674 UVHOST: server event FD is 64
2018-04-10 13:57:58,674 UVHOST: server socket FD is 65
2018-04-10 13:57:59,673 VROUTER: Retrying connection for socket 63...
2018-04-10 13:57:59,673 UVHOST: Handling connection FD 65...
2018-04-10 13:57:59,673 UVHOST: FD 65 accepted new NetLink connection FD 66
2018-04-10 13:57:59,673 DPCORE: vrouter soft reset start
2018-04-10 13:57:59,709 DPCORE: vrouter soft reset done (0)
2018-04-10 13:58:01,862 VROUTER: Adding vif 0 (gen. 1) eth device 2 PCI 0000:00:00.0 MAC 00:00:00:00:00:00 (vif MAC 00:11:ac:1d:d9:2f)
2018-04-10 13:58:01,862 VROUTER: Using 10 TX queues, 8 RX queues
2018-04-10 13:58:01,862 VROUTER: bond eth device 2 configured MAC 00:11:ac:1d:d9:2f
2018-04-10 13:58:01,862 VROUTER: bond member eth device 1 PCI 0000:83:00.1 MAC a0:36:9f:c1:99:76
2018-04-10 13:58:01,862 VROUTER: bond member eth device 1 promisc mode disabled
2018-04-10 13:58:01,862 VROUTER: bond member eth device 0 PCI 0000:03:00.0 MAC a0:36:9f:c0:0f:70
2018-04-10 13:58:01,862 VROUTER: bond member eth device 0 promisc mode disabled
2018-04-10 13:58:01,862 VROUTER: setup 8 RSS queue(s) and 0 filtering queue(s)
2018-04-10 13:58:02,037 PMD: ixgbe_dev_
2018-04-10 13:58:02,212 PMD: ixgbe_dev_
2018-04-10 13:58:02,212 VROUTER: lcore 10 TX to HW queue 0
2018-04-10 13:58:02,212 VROUTER: lcore 11 TX to HW queue 1
2018-04-10 13:58:02,212 VROUTER: lcore 12 TX to HW queue 2
2018-04-10 13:58:02,212 VROUTER: lcore 13 TX to HW queue 3
2018-04-10 13:58:02,212 VROUTER: lcore 14 TX to HW queue 4
2018-04-10 13:58:02,212 VROUTER: lcore 15 TX to HW queue 5
2018-04-10 13:58:02,212 VROUTER: lcore 16 TX to HW queue 6
2018-04-10 13:58:02,212 VROUTER: lcore 17 TX to HW queue 7
2018-04-10 13:58:02,212 VROUTER: lcore 8 TX to HW queue 8
2018-04-10 13:58:02,212 VROUTER: lcore 9 TX to HW queue 9
2018-04-10 13:58:02,212 VROUTER: lcore 10 RX from HW queue 0
2018-04-10 13:58:02,212 VROUTER: lcore 11 RX from HW queue 1
2018-04-10 13:58:02,212 VROUTER: lcore 12 RX from HW queue 2
2018-04-10 13:58:02,212 VROUTER: lcore 13 RX from HW queue 3
2018-04-10 13:58:02,212 VROUTER: lcore 14 RX from HW queue 4
2018-04-10 13:58:02,212 VROUTER: lcore 15 RX from HW queue 5
2018-04-10 13:58:02,212 VROUTER: lcore 16 RX from HW queue 6
2018-04-10 13:58:02,212 VROUTER: lcore 17 RX from HW queue 7
...
Changed in juniperopenstack: | |
importance: | Undecided → Critical |
milestone: | none → r3.2.10.0 |
milestone: | r3.2.10.0 → none |
tags: | added: 2018-0413-0535 att-aic-contrail jtac |
Changed in juniperopenstack: | |
importance: | Critical → High |
information type: | Proprietary → Private |
information type: | Private → Proprietary |
information type: | Proprietary → Private |
information type: | Private → Public |
tags: | added: blocker |