| 2018-10-03 08:57:01 |
Bernhard Koessler |
bug |
|
|
added bug |
| 2018-10-03 09:30:56 |
Bernhard Koessler |
description |
Using CC13 and Contrail 5.0.1 the procedure to restart the DPDK vrouter is
ifdown vhost0
ifup vhost0
this unbinds DPDK driver, stops/removes container, removes vhost interface and adds/starts stuff again.
However, it is seen that most of the time the scripts hangs when unbinding the DPDK driver. Container needs to be manually stopped and UIO driver unloaded in some cases to fix it.
Issuing ifdown vhost0 - unbind is stuck and shell does not return:
19:24:26 | [root@overcloud63m-compdpdk-60 ~]# ifdown vhost0
19:24:27 | INFO: rebind device 0000:06:00.0 from vfio-pci to driver i40e
19:24:27 | INFO: unbind 0000:06:00.0 from vfio-pci
19:24:55 | ^Z
19:24:55 |
19:24:55 |
19:24:56 | ^Z
19:24:56 |
19:24:56 |
19:24:56 |
19:25:58 |
19:25:58 |
19:25:58 |
19:25:58 |
19:26:01 |
19:26:01 | [1]+ Stopped ifdown vhost0
19:26:01 | [root@overcloud63m-compdpdk-60 ~]#
from another shell, stop the dpdk container (script from previous shell returns at this point - see last lines above):
19:26:01 | [root@overcloud63m-compdpdk-60 ~]# docker stop contrail-vrouter-agent-dpdk
19:26:02 | contrail-vrouter-agent-dpdk
DPDK driver unbind gets stuck and the interfaces look like this:
19:26:09 | [root@overcloud63m-compdpdk-60 ~]# /var/lib/docker/overlay2/fec97702bec79042f8007e0c53823adfb48affff0dc18235bb18c6739893a9fb/diff/opt/contrail/bin/dpdk_nic_bind.py -s
19:26:09 |
19:26:09 | Network devices using DPDK-compatible driver
19:26:09 | ============================================
19:26:09 | 0000:06:00.1 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e
19:26:09 |
19:26:09 | Network devices using kernel driver
19:26:09 | ===================================
19:26:09 | 0000:16:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=vfio-pci *Active*
19:26:09 | 0000:16:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=vfio-pci *Active*
19:26:09 | 0000:16:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=vfio-pci
19:26:09 | 0000:16:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=vfio-pci
19:26:09 | 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f0 drv=i40e unused=vfio-pci
19:26:09 | 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f1 drv=i40e unused=vfio-pci
19:26:09 |
19:26:09 | Other network devices
19:26:09 | =====================
19:26:09 | 0000:06:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
19:27:20 | [root@overcloud63m-compdpdk-60 ~]# docker stop contrail-vrouter-agent-dpdk
Also, manualy trying dpdk_nic_bind.py –u gets stuck and does not terminate and cannot be interrupted.
Sometimes the nic unbinds after killing the dpdk container but sometimes it is still stuck as shown above and UIO drier needs to be unloaded:
19:28:15 | [root@overcloud63m-compdpdk-60 ~]# modprobe -r vfio_pci
19:28:25 | [root@overcloud63m-compdpdk-60 ~]# /var/lib/docker/overlay2/fec97702bec79042f8007e0c53823adfb48affff0dc18235bb18c6739893a9fb/diff/opt/contrail/bin/dpdk_nic_bind.py -s
19:28:26 |
19:28:26 | Network devices using DPDK-compatible driver
19:28:26 | ============================================
19:28:26 | <none>
19:28:26 |
19:28:26 | Network devices using kernel driver
19:28:26 | ===================================
19:28:26 | 0000:16:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused= *Active*
19:28:26 | 0000:16:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused= *Active*
19:28:26 | 0000:16:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=
19:28:26 | 0000:16:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=
19:28:26 | 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f0 drv=i40e unused=
19:28:26 | 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f1 drv=i40e unused=
19:28:26 |
19:28:26 | Other network devices
19:28:26 | =====================
19:28:26 | 0000:06:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e
19:28:26 | 0000:06:00.1 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e
19:28:40 | [root@overcloud63m-compdpdk-60 ~]# modprobe vfio_pci
19:28:43 | [root@overcloud63m-compdpdk-60 ~]# /var/lib/docker/overlay2/fec97702bec79042f8007e0c53823adfb48affff0dc18235bb18c6739893a9fb/diff/opt/contrail/bin/dpdk_nic_bind.py -s
19:28:43 |
19:28:43 | Network devices using DPDK-compatible driver
19:28:43 | ============================================
19:28:43 | <none>
19:28:43 |
19:28:43 | Network devices using kernel driver
19:28:43 | ===================================
19:28:43 | 0000:16:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=vfio-pci *Active*
19:28:43 | 0000:16:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=vfio-pci *Active*
19:28:43 | 0000:16:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=vfio-pci
19:28:43 | 0000:16:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=vfio-pci
19:28:43 | 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f0 drv=i40e unused=vfio-pci
19:28:43 | 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f1 drv=i40e unused=vfio-pci
19:28:43 |
19:28:43 | Other network devices
19:28:43 | =====================
19:28:43 | 0000:06:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
19:28:43 | 0000:06:00.1 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
19:28:44 | [root@overcloud63m-compdpdk-60 ~]#
19:28:45 | [root@overcloud63m-compdpdk-60 ~]#
19:28:45 | [root@overcloud63m-compdpdk-60 ~]#
19:28:55 | [root@overcloud63m-compdpdk-60 ~]# ifup vhost0
19:28:55 | /usr/bin/docker-current: Error response from daemon: Conflict. The container name "/contrail-vrouter-agent-dpdk" is already in use by container a2535a9c1807937b3484cb89538d341c8d733647fa8bd9f49e8a43523ccf0b5b. You have to remove (or rename) that container to be able to reuse that name..
19:28:55 | See '/usr/bin/docker-current run --help'.
19:28:55 | Cannot find device "vhost0"
19:28:55 | Cannot find device "vhost0"
19:28:55 | ERROR : [/etc/sysconfig/network-scripts/ifup-vhost] Failed to bring up vhost0.
19:28:57 | [root@overcloud63m-compdpdk-60 ~]#
19:28:57 | [root@overcloud63m-compdpdk-60 ~]#
19:28:58 | [root@overcloud63m-compdpdk-60 ~]#
19:29:25 | [root@overcloud63m-compdpdk-60 ~]# docker rename contrail-vrouter-agent-dpdk contrail-vrouter-agent-dpdk-xx
19:29:28 | [root@overcloud63m-compdpdk-60 ~]# ifup vhost0
19:29:28 | 3962908a23ae4f75991db4f6ad5b542805706f671004286ad89bb5f7eb1c387e
19:29:28 | INFO: wait DPDK agent to run... 1
19:29:33 | INFO: wait DPDK agent to run... 2
19:29:38 | INFO: wait DPDK agent to run... 3
19:29:43 | INFO: wait DPDK agent to run... 4
19:29:48 | INFO: wait DPDK agent to run... 5
19:29:53 | INFO: wait DPDK agent to run... 6
19:29:58 | INFO: wait DPDK agent to run... 7
19:30:03 | INFO: wait DPDK agent to run... 8
19:30:08 | INFO: wait DPDK agent to run... 9
19:30:13 | INFO: wait DPDK agent to run... 10
19:30:18 | INFO: wait DPDK agent to run... 11
19:30:18 | INFO: wait vhost0 to be initilaized... 0/60
19:30:45 | INFO: wait vhost0 to be initilaized... 1/60
19:30:45 | INFO: wait vhost0 to be initilaized... 2/60
19:30:45 | INFO: vhost0 is ready. |
CC13/Contrail 5.0.1 setup with Intel X710 Fortville NICs.
Using CC13 and Contrail 5.0.1 the procedure to restart the DPDK vrouter is:
ifdown vhost0
ifup vhost0
this unbinds DPDK driver, stops/removes container, removes vhost interface and adds/starts stuff again.
However, it is seen that most of the time the scripts hangs when unbinding the DPDK driver. Container needs to be manually stopped and UIO driver unloaded in some cases to fix it.
Issuing ifdown vhost0 - unbind is stuck and shell does not return:
19:24:26 | [root@overcloud63m-compdpdk-60 ~]# ifdown vhost0
19:24:27 | INFO: rebind device 0000:06:00.0 from vfio-pci to driver i40e
19:24:27 | INFO: unbind 0000:06:00.0 from vfio-pci
19:24:55 | ^Z
19:24:55 |
19:24:55 |
19:24:56 | ^Z
19:24:56 |
19:24:56 |
19:24:56 |
19:25:58 |
19:25:58 |
19:25:58 |
19:25:58 |
19:26:01 |
19:26:01 | [1]+ Stopped ifdown vhost0
19:26:01 | [root@overcloud63m-compdpdk-60 ~]#
from another shell, stop the dpdk container (script from previous shell returns at this point - see last lines above):
19:26:01 | [root@overcloud63m-compdpdk-60 ~]# docker stop contrail-vrouter-agent-dpdk
19:26:02 | contrail-vrouter-agent-dpdk
DPDK driver unbind gets stuck and the interfaces look like this:
19:26:09 | [root@overcloud63m-compdpdk-60 ~]# /var/lib/docker/overlay2/fec97702bec79042f8007e0c53823adfb48affff0dc18235bb18c6739893a9fb/diff/opt/contrail/bin/dpdk_nic_bind.py -s
19:26:09 |
19:26:09 | Network devices using DPDK-compatible driver
19:26:09 | ============================================
19:26:09 | 0000:06:00.1 'Ethernet Controller X710 for 10GbE SFP+' drv=vfio-pci unused=i40e
19:26:09 |
19:26:09 | Network devices using kernel driver
19:26:09 | ===================================
19:26:09 | 0000:16:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=vfio-pci *Active*
19:26:09 | 0000:16:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=vfio-pci *Active*
19:26:09 | 0000:16:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=vfio-pci
19:26:09 | 0000:16:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=vfio-pci
19:26:09 | 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f0 drv=i40e unused=vfio-pci
19:26:09 | 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f1 drv=i40e unused=vfio-pci
19:26:09 |
19:26:09 | Other network devices
19:26:09 | =====================
19:26:09 | 0000:06:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
19:27:20 | [root@overcloud63m-compdpdk-60 ~]# docker stop contrail-vrouter-agent-dpdk
Also, manualy trying dpdk_nic_bind.py –u gets stuck and does not terminate and cannot be interrupted.
Sometimes the nic unbinds after killing the dpdk container but sometimes it is still stuck as shown above and UIO drier needs to be unloaded:
19:28:15 | [root@overcloud63m-compdpdk-60 ~]# modprobe -r vfio_pci
19:28:25 | [root@overcloud63m-compdpdk-60 ~]# /var/lib/docker/overlay2/fec97702bec79042f8007e0c53823adfb48affff0dc18235bb18c6739893a9fb/diff/opt/contrail/bin/dpdk_nic_bind.py -s
19:28:26 |
19:28:26 | Network devices using DPDK-compatible driver
19:28:26 | ============================================
19:28:26 | <none>
19:28:26 |
19:28:26 | Network devices using kernel driver
19:28:26 | ===================================
19:28:26 | 0000:16:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused= *Active*
19:28:26 | 0000:16:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused= *Active*
19:28:26 | 0000:16:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=
19:28:26 | 0000:16:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=
19:28:26 | 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f0 drv=i40e unused=
19:28:26 | 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f1 drv=i40e unused=
19:28:26 |
19:28:26 | Other network devices
19:28:26 | =====================
19:28:26 | 0000:06:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e
19:28:26 | 0000:06:00.1 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e
19:28:40 | [root@overcloud63m-compdpdk-60 ~]# modprobe vfio_pci
19:28:43 | [root@overcloud63m-compdpdk-60 ~]# /var/lib/docker/overlay2/fec97702bec79042f8007e0c53823adfb48affff0dc18235bb18c6739893a9fb/diff/opt/contrail/bin/dpdk_nic_bind.py -s
19:28:43 |
19:28:43 | Network devices using DPDK-compatible driver
19:28:43 | ============================================
19:28:43 | <none>
19:28:43 |
19:28:43 | Network devices using kernel driver
19:28:43 | ===================================
19:28:43 | 0000:16:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=vfio-pci *Active*
19:28:43 | 0000:16:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=vfio-pci *Active*
19:28:43 | 0000:16:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=vfio-pci
19:28:43 | 0000:16:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=vfio-pci
19:28:43 | 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f0 drv=i40e unused=vfio-pci
19:28:43 | 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=ens5f1 drv=i40e unused=vfio-pci
19:28:43 |
19:28:43 | Other network devices
19:28:43 | =====================
19:28:43 | 0000:06:00.0 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
19:28:43 | 0000:06:00.1 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,vfio-pci
19:28:44 | [root@overcloud63m-compdpdk-60 ~]#
19:28:45 | [root@overcloud63m-compdpdk-60 ~]#
19:28:45 | [root@overcloud63m-compdpdk-60 ~]#
19:28:55 | [root@overcloud63m-compdpdk-60 ~]# ifup vhost0
19:28:55 | /usr/bin/docker-current: Error response from daemon: Conflict. The container name "/contrail-vrouter-agent-dpdk" is already in use by container a2535a9c1807937b3484cb89538d341c8d733647fa8bd9f49e8a43523ccf0b5b. You have to remove (or rename) that container to be able to reuse that name..
19:28:55 | See '/usr/bin/docker-current run --help'.
19:28:55 | Cannot find device "vhost0"
19:28:55 | Cannot find device "vhost0"
19:28:55 | ERROR : [/etc/sysconfig/network-scripts/ifup-vhost] Failed to bring up vhost0.
19:28:57 | [root@overcloud63m-compdpdk-60 ~]#
19:28:57 | [root@overcloud63m-compdpdk-60 ~]#
19:28:58 | [root@overcloud63m-compdpdk-60 ~]#
19:29:25 | [root@overcloud63m-compdpdk-60 ~]# docker rename contrail-vrouter-agent-dpdk contrail-vrouter-agent-dpdk-xx
19:29:28 | [root@overcloud63m-compdpdk-60 ~]# ifup vhost0
19:29:28 | 3962908a23ae4f75991db4f6ad5b542805706f671004286ad89bb5f7eb1c387e
19:29:28 | INFO: wait DPDK agent to run... 1
19:29:33 | INFO: wait DPDK agent to run... 2
19:29:38 | INFO: wait DPDK agent to run... 3
19:29:43 | INFO: wait DPDK agent to run... 4
19:29:48 | INFO: wait DPDK agent to run... 5
19:29:53 | INFO: wait DPDK agent to run... 6
19:29:58 | INFO: wait DPDK agent to run... 7
19:30:03 | INFO: wait DPDK agent to run... 8
19:30:08 | INFO: wait DPDK agent to run... 9
19:30:13 | INFO: wait DPDK agent to run... 10
19:30:18 | INFO: wait DPDK agent to run... 11
19:30:18 | INFO: wait vhost0 to be initilaized... 0/60
19:30:45 | INFO: wait vhost0 to be initilaized... 1/60
19:30:45 | INFO: wait vhost0 to be initilaized... 2/60
19:30:45 | INFO: vhost0 is ready. |
|
| 2018-10-03 12:09:30 |
Florin Vinti |
bug |
|
|
added subscriber Florin Vinti |
| 2018-10-03 16:23:28 |
Vinod Nair |
juniperopenstack: assignee |
Vinod Nair (vinodnair) |
Bernhard Koessler (bkoessler) |
|
| 2018-10-03 17:04:47 |
Bernhard Koessler |
juniperopenstack: assignee |
Bernhard Koessler (bkoessler) |
Vinod Nair (vinodnair) |
|
| 2018-10-08 18:09:15 |
Jeba Paulaiyan |
nominated for series |
|
juniperopenstack/r5.0 |
|
| 2018-10-08 18:09:15 |
Jeba Paulaiyan |
bug task added |
|
juniperopenstack/r5.0 |
|
| 2018-10-08 18:09:15 |
Jeba Paulaiyan |
nominated for series |
|
juniperopenstack/trunk |
|
| 2018-10-08 18:09:15 |
Jeba Paulaiyan |
bug task added |
|
juniperopenstack/trunk |
|
| 2018-10-08 18:09:20 |
Jeba Paulaiyan |
juniperopenstack/r5.0: importance |
Undecided |
Critical |
|
| 2018-10-08 18:09:27 |
Jeba Paulaiyan |
juniperopenstack/r5.0: assignee |
|
Vinod Nair (vinodnair) |
|
| 2018-10-08 18:09:31 |
Jeba Paulaiyan |
juniperopenstack/r5.0: milestone |
|
r5.0.2 |
|
| 2018-10-08 18:09:33 |
Jeba Paulaiyan |
juniperopenstack/trunk: milestone |
|
r5.1.0 |
|
| 2018-10-08 22:34:29 |
Vinod Nair |
juniperopenstack/r5.0: assignee |
Vinod Nair (vinodnair) |
Michael Henkel (mhenkel-3) |
|
| 2018-10-08 22:35:02 |
Vinod Nair |
juniperopenstack/trunk: assignee |
Vinod Nair (vinodnair) |
Michael Henkel (mhenkel-3) |
|
| 2018-10-09 11:08:54 |
Sivakumar Ganapathy |
tags |
cc13 dpdk vrouter |
cc13 dpdk |
|
| 2018-10-09 20:41:32 |
Jeba Paulaiyan |
juniperopenstack/r5.0: assignee |
Michael Henkel (mhenkel-3) |
alexey-mr (alexey-morlang) |
|
| 2018-10-09 20:41:43 |
Jeba Paulaiyan |
juniperopenstack/trunk: assignee |
Michael Henkel (mhenkel-3) |
alexey-mr (alexey-morlang) |
|
| 2018-10-09 20:42:00 |
Jeba Paulaiyan |
tags |
cc13 dpdk |
blocker cc13 dpdk |
|
| 2018-10-11 15:50:36 |
alexey-mr |
juniperopenstack/r5.0: status |
New |
Incomplete |
|
| 2018-10-11 15:50:38 |
alexey-mr |
juniperopenstack/trunk: status |
New |
Incomplete |
|
| 2018-10-23 21:58:26 |
Jeba Paulaiyan |
juniperopenstack/r5.0: importance |
Critical |
High |
|
| 2018-10-23 21:58:29 |
Jeba Paulaiyan |
juniperopenstack/trunk: importance |
Critical |
High |
|
| 2018-10-23 21:58:42 |
Jeba Paulaiyan |
tags |
blocker cc13 dpdk |
cc13 dpdk |
|
| 2018-10-23 21:58:46 |
Jeba Paulaiyan |
juniperopenstack/r5.0: milestone |
r5.0.2 |
|
|
| 2018-10-23 21:58:48 |
Jeba Paulaiyan |
juniperopenstack/trunk: milestone |
r5.1.0 |
|
|
| 2018-11-27 14:19:06 |
alexey-mr |
juniperopenstack/r5.0: status |
Incomplete |
Won't Fix |
|
| 2018-11-27 14:19:10 |
alexey-mr |
juniperopenstack/trunk: status |
Incomplete |
Won't Fix |
|
