Host OpenVSwitch+DPDK process can segfault from a VM running DPDK-enabled application

Bug #1615012 reported by Mikhail Chernik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Maria Zlatkova

Bug Description

Description:

Due to a bug in DPDK prior to 16.04, a dpdk-enabled process running in VM can crash host dpdk-enabled OpenVSwitch process. Detailed description can be found here: http://www.dpdk.org/ml/archives/dev/2016-May/039422.html

Steps to reproduce:

1. Enable experimental features on MOS 9.0 Fuel Master
2. Deploy a compute node with configured DPDK
3. Create a VM on the compute node
4. Download and compile DPDK ver <= 2.2 on the VM
5. Follow instructions from link above

Expected result:
Host OVS process is not affected

Actual result:
Host OVS crashes with segfault after second port initialization

OVS+DPDK pmd segfault on host also could be reproduces next way:

1. Enable experimental features on MOS 9.0 Fuel Master
2. Deploy a compute node with configured DPDK
3. Create a VM on the compute node. Add one extra interface.
4. Download and compile DPDK
5. Download DPDK-based application. 100% reproducebility achieved with pktgen-dpdk. http://dpdk.org/browse/apps/pktgen-dpdk
6. Start DPDk-based application with extra interface attached to DPDK on guest.
7. Enable high workload traffic on the interface that attached to DPDK on guest.
8. Kill DPDK-based application
9. Start DPDK-based application immediately.

Expected result:
Host OVS process is not affected

Actual result:
Host OVS crashes with segfault after DPDK-based app init.

Sergey Matov (smatov)
Changed in fuel:
status: New → In Progress
Changed in fuel:
importance: Undecided → High
Revision history for this message
Sergey Matov (smatov) wrote :

Described bug is partly fixed by
https://review.fuel-infra.org/#/c/25699/
Re-init of testpmd ports does not cause segfault anymore.

For the situation where DPDK-based application on guest stops incorrectly(e.g. killing pid) and re-starts after we are awaiting fix in new versions of DPDK. To avoid segfault of host vswitchd process segfault after re-starting guest DPDK-based app HugePages for guest application should be freed(or re-mounted).

Appropriate information will be added to the release notes.

Sergey Matov (smatov)
description: updated
Sergey Matov (smatov)
tags: added: release-notes
Revision history for this message
Sergey Matov (smatov) wrote :

Release notes

 - Referring to bug https://bugs.launchpad.net/fuel/+bug/1615012 for DPDK-accelerated compute node if DPDK-based application on guest stops without freeing HugePages it is recommended to re-mount HugePages on guest VM to avoid OVS-DPDK segmentation fault on compute node.

Changed in fuel:
assignee: Sergey Matov (smatov) → Fuel Documentation Team (fuel-docs)
tags: added: release-notes-done
removed: release-notes
Changed in fuel:
assignee: Fuel Documentation Team (fuel-docs) → Maria Zlatkova (mzlatkova)
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.