DPDK: Agent stuck in init state; Error connecting to socket /var/run/vrouter/dpdk_pkt0

[amit surana]

3.0.2 b44

Agent is continuously crashing. dmesg:

[602007.010437] ------------[ cut here ]------------
[602007.010445] WARNING: CPU: 8 PID: 6138 at /build/linux-Mxzr_W/linux-3.13.0/net/core/dev.c:5590 rollback_registered_many+0x7e/0x2d0()
[602007.010446] Modules linked in: rte_kni(OX) igb_uio(OX) vrouter(OX) joydev hid_generic usbhid hid xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables uio nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ses enclosure bonding gpio_ich nfsd dm_multipath scsi_dh ttm drm_kms_helper drm syscopyarea sysfillrect sysimgblt intel_rapl x86_pkg_temp_thermal intel_powerclamp auth_rpcgss coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd lpc_ich ipmi_si sb_edac shpchp edac_core mac_hid nfs_acl nfs lockd sunrpc fscache lp parport mpt2sas raid_class isci igb ixgbe i2c_algo_bit ahci libsas dca ptp libahci scsi_transport_sas pps_core mdio [last unloaded: igb_uio]
[602007.010500] CPU: 8 PID: 6138 Comm: contrail-vroute Tainted: G W OX 3.13.0-85-generic #129-Ubuntu
[602007.010501] Hardware name: Quanta S210-X22RQ/S210-X22RQ, BIOS S2RQ3A20 01/25/2013
[602007.010503] 0000000000000000 ffff88049d531cd8 ffffffff8172b6a7 0000000000000000
[602007.010507] 0000000000000009 ffff88049d531d10 ffffffff810699cd ffff881429cb3000
[602007.010510] ffff88049d531d08 ffff88049d531d78 ffff88049d531d38 0000000000000001
[602007.010513] Call Trace:
[602007.010520] [<ffffffff8172b6a7>] dump_stack+0x64/0x82
[602007.010524] [<ffffffff810699cd>] warn_slowpath_common+0x7d/0xa0
[602007.010526] [<ffffffff81069aaa>] warn_slowpath_null+0x1a/0x20
[602007.010529] [<ffffffff8162b2ae>] rollback_registered_many+0x7e/0x2d0
[602007.010531] [<ffffffff8162b52d>] rollback_registered+0x2d/0x40
[602007.010534] [<ffffffff8162be58>] unregister_netdevice_queue+0x58/0xc0
[602007.010537] [<ffffffff8162bedc>] unregister_netdev+0x1c/0x30
[602007.010546] [<ffffffffa05cb2e8>] kni_dev_remove+0xb8/0x360 [rte_kni]
[602007.010552] [<ffffffffa05cbe4a>] kni_ioctl_create+0x56a/0x890 [rte_kni]
[602007.010558] [<ffffffffa05cc1a8>] kni_ioctl+0x38/0x40 [rte_kni]
[602007.010561] [<ffffffff811d5050>] do_vfs_ioctl+0x2e0/0x4c0
[602007.010566] [<ffffffff811c1b0c>] ? vfs_write+0x15c/0x1f0
[602007.010568] [<ffffffff811d52b1>] SyS_ioctl+0x81/0xa0
[602007.010571] [<ffffffff811c24cc>] ? SyS_write+0x7c/0xa0
[602007.010575] [<ffffffff8173c39d>] system_call_fastpath+0x1a/0x1f
[602007.010576] ---[ end trace 31b70eb5dfeb2a80 ]---
[602007.077803] network todo 'vhost0' but state 0
[602007.080014] CPU: 8 PID: 6138 Comm: contrail-vroute Tainted: G W OX 3.13.0-85-generic #129-Ubuntu
[602007.080015] Hardware name: Quanta S210-X22RQ/S210-X22RQ, BIOS S2RQ3A20 01/25/2013
[602007.080016] 0000000000000000 ffff88049d531d40 ffffffff8172b6a7 ffff881429cb3000
[602007.080020] ffff881429cb33e0 ffff88049d531d98 ffffffff8163264a 0000000000000000
[602007.080022] ffff881429cb3000 ffff88049d531d60 ffff88049d531d60 ffff881429cb3000
[602007.080025] Call Trace:
[602007.080028] [<ffffffff8172b6a7>] dump_stack+0x64/0x82
[602007.080032] [<ffffffff8163264a>] netdev_run_todo+0x2aa/0x2f0
[602007.080036] [<ffffffff8163c80e>] rtnl_unlock+0xe/0x10
[602007.080038] [<ffffffff8162bee1>] unregister_netdev+0x21/0x30
[602007.080045] [<ffffffffa05cb2e8>] kni_dev_remove+0xb8/0x360 [rte_kni]
[602007.080051] [<ffffffffa05cbe4a>] kni_ioctl_create+0x56a/0x890 [rte_kni]
[602007.080057] [<ffffffffa05cc1a8>] kni_ioctl+0x38/0x40 [rte_kni]
[602007.080059] [<ffffffff811d5050>] do_vfs_ioctl+0x2e0/0x4c0
[602007.080061] [<ffffffff811c1b0c>] ? vfs_write+0x15c/0x1f0
[602007.080063] [<ffffffff811d52b1>] SyS_ioctl+0x81/0xa0
[602007.080065] [<ffffffff811c24cc>] ? SyS_write+0x7c/0xa0
[602007.080068] [<ffffffff8173c39d>] system_call_fastpath+0x1a/0x1f
[602007.587144] contrail-vroute[6138]: segfault at c ip 00000000004f3d31 sp 00007f42818408b0 error 4 in contrail-vrouter-dpdk[400000+187000]
[602009.372921] KNI: /dev/kni closed
[602069.054348] KNI: Multiple kernel thread mode enabled
[602069.054353] KNI: /dev/kni opened
[602069.054638] KNI: Creating kni...
[602069.054654] KNI: tx_phys: 0x0000000054d9e780, tx_q addr: 0xffff880054d9e780
[602069.054656] KNI: rx_phys: 0x0000000054d9c700, rx_q addr: 0xffff880054d9c700
[602069.054657] KNI: alloc_phys: 0x0000000054d9a680, alloc_q addr: 0xffff880054d9a680
[602069.054658] KNI: free_phys: 0x0000000054d98600, free_q addr: 0xffff880054d98600
[602069.054659] KNI: req_phys: 0x0000000054d96580, req_q addr: 0xffff880054d96580
[602069.054660] KNI: resp_phys: 0x0000000054d94500, resp_q addr: 0xffff880054d94500
[602069.054661] KNI: mbuf_phys: 0x000000005817dec0, mbuf_kva: 0xffff88005817dec0
[602069.054661] KNI: mbuf_va: 0x00007f8cb4b7dec0
[602069.054662] KNI: mbuf_size: 9216
[602069.054667] KNI: pci_bus: 03:00:00
[602069.094374] igb_uio 0000:03:00.0: (PCI Express:5.0GT/s:Width x8)
[602069.094376] 90:e2:ba:b8:70:c4
[602069.094460] igb_uio 0000:03:00.0 (unregistered net_device): MAC: 2, PHY: 0, PBA No: E68793-006
[602069.094462] igb_uio 0000:03:00.0 (unregistered net_device): Enabled Features: RxQ: 1 TxQ: 1
[602069.094473] igb_uio 0000:03:00.0 (unregistered net_device): Intel(R) 10 Gigabit Network Connection
[602069.094478] KNI: Error: error -17 registering device "vhost0"

[Raja Sivaramakrishnan (raja-u)]

Not seen any more.