dpdk eal initialisation takes upwards of 2 minutes during restart

Bug #1793729 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Neutron Open vSwitch Charm
Fix Released
Medium
James Page

Bug Description

During restarts of ovs, the dpdk EAL init takes a long time:

2018-08-29T16:43:28.945Z|00016|dpdk|INFO|EAL: Probing VFIO support...
2018-08-29T16:45:45.363Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0
2018-08-29T16:45:45.363Z|00018|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2018-08-29T16:45:45.363Z|00019|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0
2018-08-29T16:45:45.363Z|00020|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2018-08-29T16:45:45.363Z|00021|dpdk|INFO|EAL: PCI device 0000:03:00.0 on NUMA socket 0
2018-08-29T16:45:45.363Z|00022|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe
2018-08-29T16:45:45.363Z|00023|dpdk|INFO|EAL: PCI device 0000:03:00.1 on NUMA socket 0
2018-08-29T16:45:45.363Z|00024|dpdk|INFO|EAL: probe driver: 8086:154d net_ixgbe
2018-08-29T16:45:45.363Z|00025|dpdk|INFO|EAL: PCI device 0000:05:00.0 on NUMA socket 0
2018-08-29T16:45:45.363Z|00026|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb
2018-08-29T16:45:45.363Z|00027|dpdk|INFO|EAL: PCI device 0000:05:00.1 on NUMA socket 0
2018-08-29T16:45:45.363Z|00028|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb
2018-08-29T16:45:45.363Z|00029|dpdk|INFO|DPDK pdump packet capture enabled
2018-08-29T16:45:45.370Z|00030|dpdk|INFO|DPDK Enabled - initialized

This can be reduced by passing the pci-whitelist configuration to OVS, so that only whitelisted devices are scanned for VFIO support.

James Page (james-page)
Changed in charm-neutron-openvswitch:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → James Page (james-page)
milestone: none → 18.11
James Page (james-page)
Changed in charm-neutron-openvswitch:
milestone: 18.11 → 19.04
David Ames (thedac)
Changed in charm-neutron-openvswitch:
milestone: 19.04 → 19.07
Revision history for this message
James Page (james-page) wrote :

Note we can whitelist the devices that OVS will probe which will improve this problem.

Changed in charm-neutron-openvswitch:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-openvswitch (master)

Fix proposed to branch: master
Review: https://review.opendev.org/667046

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-openvswitch (master)

Reviewed: https://review.opendev.org/667046
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=3c66c48bbd11922aa4740e752dfdb5e4f35a959d
Submitter: Zuul
Branch: master

commit 3c66c48bbd11922aa4740e752dfdb5e4f35a959d
Author: James Page <email address hidden>
Date: Mon Jun 24 10:11:04 2019 +0100

    dpdk: misc fixes for EAL initialization

    For OpenStack Queens and later instances are setup with vhostuser
    ports in server mode, with the OVS side of the port connecting
    as a client to the vhostuser socket on disk; as a result we no
    longer need to pass permissions information via dpdk-extra (the
    two passed options are not valid in later OVS versions).

    In addition, we should also whitelist the devices we're going to
    use; this ensures that EAL initialization does not take an extended
    period of time reducing the amount of time taken to restart OVS.

    Change-Id: I224e778de0ed6e279b2de7f4f46781df33121165
    Closes-Bug: 1833734
    Closes-Bug: 1793729

Changed in charm-neutron-openvswitch:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-openvswitch (stable/19.04)

Fix proposed to branch: stable/19.04
Review: https://review.opendev.org/668635

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-openvswitch (stable/19.04)

Reviewed: https://review.opendev.org/668635
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=bd354841d5422f5d936c9f031228c3d5889368df
Submitter: Zuul
Branch: stable/19.04

commit bd354841d5422f5d936c9f031228c3d5889368df
Author: James Page <email address hidden>
Date: Mon Jun 24 10:11:04 2019 +0100

    dpdk: misc fixes for EAL initialization

    For OpenStack Queens and later instances are setup with vhostuser
    ports in server mode, with the OVS side of the port connecting
    as a client to the vhostuser socket on disk; as a result we no
    longer need to pass permissions information via dpdk-extra (the
    two passed options are not valid in later OVS versions).

    In addition, we should also whitelist the devices we're going to
    use; this ensures that EAL initialization does not take an extended
    period of time reducing the amount of time taken to restart OVS.

    Change-Id: I224e778de0ed6e279b2de7f4f46781df33121165
    Closes-Bug: 1833734
    Closes-Bug: 1793729
    (cherry picked from commit 3c66c48bbd11922aa4740e752dfdb5e4f35a959d)

James Page (james-page)
Changed in charm-neutron-openvswitch:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.