Comment 0 for bug 1909062

Revision history for this message
Manish Chopra (chopramanish1988) wrote : Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure

Customer is reporting a problem with QL41xxx and Ubuntu internal DNS server. The issue appeared when the customer updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux.

Problem Definition:

Product: PowerEdge R740xd
Serial: C7J90W2
Hostname: xkubmin1r12
OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic
NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02)
Firmware: 15.15.11
Inbox driver qede v8.37.0.20

Completed Detailed Problem Description:
Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs.

Customer is experiencing the same issue described in this article.
https://github.com/kubernetes/kubernetes/issues/95365

Customer Impact: Production site
The QLogic Nic 41262 is their main nic for all of their 14G environment thousands of servers. Unclear how many of those hosts are Kubernetes, but the point is they want the QL41000 to work since it is very prevalent in the entire environment.

Below patch recently on upstream fixes this -
[Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel]

commit 5d5647dad259bb416fd5d3d87012760386d97530
Author: Manish Chopra <email address hidden>
Date: Mon Dec 21 06:55:30 2020 -0800

    qede: fix offload for IPIP tunnel packets

    IPIP tunnels packets are unknown to device,
    hence these packets are incorrectly parsed and
    caused the packet corruption, so disable offlods
    for such packets at run time.

    Signed-off-by: Manish Chopra <email address hidden>
    Signed-off-by: Sudarsana Kalluru <email address hidden>
    Signed-off-by: Igor Russkikh <email address hidden>
    Link: https://<email address hidden>
    Signed-off-by: Jakub Kicinski <email address hidden>

Thanks,
Manish