datapath: Add missing case OVS_TUNNEL_KEY_ATTR_PAD

Bug #1676679 reported by bugproxy on 2017-03-28
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Undecided
Unassigned
linux (Ubuntu)
Medium
Canonical Kernel Team
Yakkety
Medium
Joseph Salisbury
Zesty
Medium
Joseph Salisbury
openvswitch (Ubuntu)
Undecided
Taco Screen team
Yakkety
Undecided
Unassigned
Zesty
Undecided
Unassigned

Bug Description

---Problem Description---
Recreate and error info:

Hit a new issue with OVS after updating to the Ubuntu 4.8 kernel from
the Ubuntu 4.4 kernel.

Iperf was used to send traffic between client VMs over VXLAN. The
traffic did still flow, but every packet had to go to user space due to
the flow creation failures, which drastically impacted performance and
cpu utilization.

When using VXLAN, the following error is showing up in dmesg

openvswitch: netlink: Unknown IP tunnel attribute 14

Also there are tons of these errors in the openvswitch log

2017-03-01T15:50:47.860Z|00018|dpif(handler164)|WARN|system@ovs-system:
failed to put[create] (Invalid argument)
ufid:2d1a9aeb-7b24-4235-a208-a01f98237e60 recirc_id(0),dp_hash(0/0),skb_pri

Debug showed that this attribute, OVS_TUNNEL_KEY_ATTR_PAD, was being
seen in the switch statement in method, static int ip_tun_from_nlattr,
in flow_netlink.c . Because there is no case for this attribute, the
default is hit and returns an error.

The issue was first seen using the packages in the Ubuntu 4.8 kernel,
which is OVS 2.5. OVS 2.6 and 2.6.1 were also tried with the kernel
packages and the same issue was seen. Tried building OVS 2.7 and
loading the openvswitch-datapath-dkms_2.7.0-1_all.deb that got built but
the issue persisted. The proposed patch seems to eliminate the error
messages and also fixed the segmentation and performance issues that
were seen.

---uname output---
stock 4.8 kernel

Machine Type = p8

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 Hit a new issue with OVS after updating to the Ubuntu 4.8 kernel from
the Ubuntu 4.4 kernel.

Iperf was used to send traffic between client VMs over VXLAN. The
traffic did still flow, but every packet had to go to user space due to
the flow creation failures, which drastically impacted performance and
cpu utilization.

When using VXLAN, the following error is showing up in dmesg

openvswitch: netlink: Unknown IP tunnel attribute 14

Link to the patch is https://patchwork.ozlabs.org/patch/738856/

Default Comment by Bridge

tags: added: architecture-ppc64 bugnameltc-152930 severity-medium targetmilestone-inin1610
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
affects: ubuntu → openvswitch (Ubuntu)
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu):
status: New → Confirmed

------- Comment From <email address hidden> 2017-05-08 11:33 EDT-------
Any updates on whether or not this will make Yakkety?

bugproxy (bugproxy) on 2017-05-08
tags: added: targetmilestone-inin16042
removed: targetmilestone-inin1610

Patch looks to have landed in the upstream kernel as of 4.11-rc4. This should be considered for SRU for Zesty (4.10) and Yakkety (4.8). I've marked it Fix Committed for Artful.

$ git describe --contains 8f3dbfd79ed9ef9770305a7cc4e13dfd31ad2cd0
v4.11-rc4~28^2~49

$ git show 8f3dbfd79ed9ef9770305a7cc4e13dfd31ad2cd0
commit 8f3dbfd79ed9ef9770305a7cc4e13dfd31ad2cd0
Author: Kris Murphy <email address hidden>
Date: Thu Mar 16 10:51:28 2017 -0500

    openvswitch: Add missing case OVS_TUNNEL_KEY_ATTR_PAD

Changed in linux (Ubuntu Zesty):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in linux (Ubuntu Yakkety):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in linux (Ubuntu):
status: New → Fix Committed
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → Medium
Changed in linux (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in linux (Ubuntu Zesty):
importance: Undecided → Medium
Changed in linux (Ubuntu Yakkety):
status: New → Triaged
Changed in linux (Ubuntu Zesty):
status: New → Triaged
Changed in linux (Ubuntu Yakkety):
status: Triaged → In Progress
Changed in linux (Ubuntu Zesty):
status: Triaged → In Progress
Changed in linux (Ubuntu Yakkety):
assignee: Canonical Kernel Team (canonical-kernel-team) → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Zesty):
assignee: Canonical Kernel Team (canonical-kernel-team) → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Zesty):
status: In Progress → Fix Released
Joseph Salisbury (jsalisbury) wrote :

Commit 8f3dbfd79ed9 was added to Zesty in version: Ubuntu-4.10.0-16.

However, Yakkety does not yet have this commit. I built a Yakkety test kernel with a pick of commit 8f3dbfd79ed9. It can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1676679/

Can you test this kernel and see if it resolves this bug?

Thanks in advance!

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-05-17 10:01 EDT-------
The headers generic package for ppc64el has x86 binaries in it so it breaks dkms for us so its hard to test.

Joseph Salisbury (jsalisbury) wrote :

Would you be able to run the make scripts inside the headers folder on the target machine? That would rebuild the x86 tools as ppc.

This make usually fails at some point. but it rebuilds the necessary tools that are necessary to build a module.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-05-18 10:10 EDT-------
works, thanks!

tags: added: verification-done-yakkety
Joseph Salisbury (jsalisbury) wrote :

Thanks for the update. I have not submitted an SRU request yet, so I removed the verification-done tag you added. That testing will be requested when the fix is in -proposed.

Thanks!

tags: removed: verification-done-yakkety
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu Yakkety):
status: New → Confirmed
Changed in openvswitch (Ubuntu Zesty):
status: New → Confirmed
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Changed in ubuntu-power-systems:
status: New → In Progress
James Page (james-page) wrote :

The openvswitch package no long ships any datapath components; marking openvswitch tasks as invalid.

Changed in openvswitch (Ubuntu Yakkety):
status: Confirmed → Invalid
Changed in openvswitch (Ubuntu Zesty):
status: Confirmed → Invalid
Changed in openvswitch (Ubuntu):
status: Confirmed → Invalid
Juerg Haefliger (juergh) on 2017-06-09
Changed in linux (Ubuntu Yakkety):
status: In Progress → Fix Committed
Changed in ubuntu-power-systems:
status: In Progress → Fix Committed

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag 'verification-needed-yakkety' to 'verification-failed-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-yakkety
Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
bugproxy (bugproxy) on 2017-06-15
tags: added: verification-done-yakkety
removed: verification-needed-yakkety
Launchpad Janitor (janitor) wrote :
Download full text (3.8 KiB)

This bug was fixed in the package linux - 4.8.0-58.63

---------------
linux (4.8.0-58.63) yakkety; urgency=low

  * linux: 4.8.0-58.63 -proposed tracker (LP: #1700533)

  * CVE-2017-1000364
    - Revert "UBUNTU: SAUCE: mm: Only expand stack if guard area is hit"
    - Revert "mm: do not collapse stack gap into THP"
    - Revert "mm: enlarge stack guard gap"
    - mm: vma_adjust: remove superfluous confusing update in remove_next == 1 case
    - mm: larger stack guard gap, between vmas
    - mm: fix new crash in unmapped_area_topdown()
    - Allow stack to grow up to address space limit

linux (4.8.0-57.62) yakkety; urgency=low

  * linux: 4.8.0-57.62 -proposed tracker (LP: #1699035)

  * CVE-2017-1000364
    - SAUCE: mm: Only expand stack if guard area is hit

  * CVE-2017-7374
    - fscrypt: remove broken support for detecting keyring key revocation

  * CVE-2017-100363
    - char: lp: fix possible integer overflow in lp_setup()

  * CVE-2017-9242
    - ipv6: fix out of bound writes in __ip6_append_data()

  * CVE-2017-9075
    - sctp: do not inherit ipv6_{mc|ac|fl}_list from parent

  * CVE-2017-9074
    - ipv6: Prevent overrun when parsing v6 header options

  * CVE-2017-9076
    - ipv6/dccp: do not inherit ipv6_mc_list from parent

  * CVE-2017-9077
    - ipv6/dccp: do not inherit ipv6_mc_list from parent

  * CVE-2017-8890
    - dccp/tcp: do not inherit mc_list from parent

  * extend-diff-ignore should use exact matches (LP: #1693504)
    - [Packaging] exact extend-diff-ignore matches

  * APST quirk needed for Intel NVMe (LP: #1686592)
    - nvme: Quirk APST on Intel 600P/P3100 devices

  * regression: the 4.8 hwe kernel does not create the
    /sys/block/*/device/enclosure_device:* symlinks (LP: #1691899)
    - scsi: ses: Fix SAS device detection in enclosure

  * datapath: Add missing case OVS_TUNNEL_KEY_ATTR_PAD (LP: #1676679)
    - openvswitch: Add missing case OVS_TUNNEL_KEY_ATTR_PAD

  * connection flood to port 445 on mounting cifs volume under kernel
    (LP: #1686099)
    - cifs: Do not send echoes before Negotiate is complete

  * Support IPMI system interface on Cavium ThunderX (LP: #1688132)
    - i2c: octeon: Rename driver to prepare for split
    - i2c: octeon: Split the driver into two parts
    - [Config] CONFIG_I2C_THUNDERX=m
    - i2c: thunderx: Add i2c driver for ThunderX SOC
    - i2c: thunderx: Add SMBUS alert support
    - i2c: octeon,thunderx: Move register offsets to struct
    - i2c: octeon: Sort include files alphabetically
    - i2c: octeon: Use booleon values for booleon variables
    - i2c: octeon: thunderx: Add MAINTAINERS entry
    - i2c: octeon: Fix set SCL recovery function
    - i2c: octeon: Avoid sending STOP during recovery
    - i2c: octeon: Fix high-level controller status check
    - i2c: octeon: thunderx: TWSI software reset in recovery
    - i2c: octeon: thunderx: Remove double-check after interrupt
    - i2c: octeon: thunderx: Limit register access retries
    - i2c: thunderx: Enable HWMON class probing

  * CVE-2017-5577
    - drm/vc4: Return -EINVAL on the overflow checks failing.

  * Merlin SGMII fail on Ubuntu Xenial HWE kernel (LP: #1686305)
    - net: phy: marvell: fix Marvell 88E1512 u...

Read more...

Changed in linux (Ubuntu Yakkety):
status: Fix Committed → Fix Released
bugproxy (bugproxy) on 2019-05-03
tags: added: targetmilestone-inin1610
removed: targetmilestone-inin16042
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers