NETDEV WATCHDOG: eno49 (ixgbe): transmit queue 34 timed out

Bug #1943879 reported by Satish Patel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

I am running ubuntu Focal version with following kernel and NIC and noticed where strange error in kernel logs and my nic went down, does any one know about this?

# uname -a
Linux ostack-phx-comp-gen-1-27.v1v0x.net 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64x

# lspci | grep -i eth
06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)

[Thu Sep 16 01:43:51 2021] ------------[ cut here ]------------
[Thu Sep 16 01:43:51 2021] NETDEV WATCHDOG: eno49 (ixgbe): transmit queue 34 timed out
[Thu Sep 16 01:43:51 2021] WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x258/0x260
[Thu Sep 16 01:43:51 2021] Modules linked in: nf_conntrack_netlink ebt_arp nft_compat nf_tables_set nft_meta_bridgel
[Thu Sep 16 01:43:51 2021] ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_mi
[Thu Sep 16 01:43:51 2021] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 5.4.0-42-generic #46-Ubuntu
[Thu Sep 16 01:43:51 2021] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017
[Thu Sep 16 01:43:51 2021] RIP: 0010:dev_watchdog+0x258/0x260
[Thu Sep 16 01:43:51 2021] Code: 85 c0 75 e5 eb 9f 4c 89 ff c6 05 1f f5 e7 00 01 e8 8d bb fa ff 44 89 e9 4c 89 fe 47
[Thu Sep 16 01:43:51 2021] RSP: 0018:ffffb92d466b4e30 EFLAGS: 00010286
[Thu Sep 16 01:43:51 2021] RAX: 0000000000000000 RBX: ffff9373ef57cec0 RCX: 000000000000083f
[Thu Sep 16 01:43:51 2021] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
[Thu Sep 16 01:43:51 2021] RBP: ffffb92d466b4e60 R08: ffff937bdf8d78c8 R09: 0000000000000004
[Thu Sep 16 01:43:51 2021] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000040
[Thu Sep 16 01:43:51 2021] R13: 0000000000000022 R14: ffff9373ef580480 R15: ffff9373ef580000
[Thu Sep 16 01:43:51 2021] FS: 0000000000000000(0000) GS:ffff937bdf8c0000(0000) knlGS:0000000000000000
[Thu Sep 16 01:43:51 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Sep 16 01:43:51 2021] CR2: 00007f3697ec19dc CR3: 000000103ce0a001 CR4: 00000000001626e0
[Thu Sep 16 01:43:51 2021] Call Trace:
[Thu Sep 16 01:43:51 2021] <IRQ>
[Thu Sep 16 01:43:51 2021] ? pfifo_fast_enqueue+0x150/0x150
[Thu Sep 16 01:43:51 2021] call_timer_fn+0x32/0x130
[Thu Sep 16 01:43:51 2021] __run_timers.part.0+0x180/0x280
[Thu Sep 16 01:43:51 2021] ? tick_sched_handle+0x33/0x60
[Thu Sep 16 01:43:51 2021] ? tick_sched_timer+0x3d/0x80
[Thu Sep 16 01:43:51 2021] ? ktime_get+0x3e/0xa0
[Thu Sep 16 01:43:51 2021] run_timer_softirq+0x2a/0x50
[Thu Sep 16 01:43:51 2021] __do_softirq+0xe1/0x2d6
[Thu Sep 16 01:43:51 2021] ? hrtimer_interrupt+0x13b/0x220
[Thu Sep 16 01:43:51 2021] irq_exit+0xae/0xb0
[Thu Sep 16 01:43:51 2021] smp_apic_timer_interrupt+0x7b/0x140
[Thu Sep 16 01:43:51 2021] apic_timer_interrupt+0xf/0x20
[Thu Sep 16 01:43:51 2021] </IRQ>
[Thu Sep 16 01:43:51 2021] RIP: 0010:cpuidle_enter_state+0xc5/0x450
[Thu Sep 16 01:43:51 2021] Code: ff e8 bf 08 81 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 65 03 00 0d
[Thu Sep 16 01:43:51 2021] RSP: 0018:ffffb92d4634fe38 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[Thu Sep 16 01:43:51 2021] RAX: ffff937bdf8ead00 RBX: ffffffff8d159c00 RCX: 000000000000001f
[Thu Sep 16 01:43:51 2021] RDX: 0000000000000000 RSI: 000000003342629e RDI: 0000000000000000
[Thu Sep 16 01:43:51 2021] RBP: ffffb92d4634fe78 R08: 0011dc129ea2a1f1 R09: 0011dc17357faf00
[Thu Sep 16 01:43:51 2021] R10: ffff937bdf8e9a00 R11: ffff937bdf8e99e0 R12: ffffd9253fac20a8
[Thu Sep 16 01:43:51 2021] R13: 0000000000000004 R14: 0000000000000004 R15: ffffd9253fac20a8
[Thu Sep 16 01:43:51 2021] ? cpuidle_enter_state+0xa1/0x450
[Thu Sep 16 01:43:51 2021] cpuidle_enter+0x2e/0x40
[Thu Sep 16 01:43:51 2021] call_cpuidle+0x23/0x40
[Thu Sep 16 01:43:51 2021] do_idle+0x1dd/0x270
[Thu Sep 16 01:43:51 2021] cpu_startup_entry+0x20/0x30
[Thu Sep 16 01:43:51 2021] start_secondary+0x167/0x1c0
[Thu Sep 16 01:43:51 2021] secondary_startup_64+0xa4/0xb0
[Thu Sep 16 01:43:51 2021] ---[ end trace cb80e9f61341ace0 ]---
[Thu Sep 16 01:43:51 2021] ixgbe 0000:06:00.0 eno49: initiating reset due to tx timeout
[Thu Sep 16 01:43:51 2021] ixgbe 0000:06:00.0 eno49: Reset adapter
[Thu Sep 16 01:43:51 2021] ixgbe 0000:06:00.0 eno49: TXDCTL.ENABLE for one or more queues not cleared within the pod
[Thu Sep 16 01:43:51 2021] ixgbe 0000:06:00.0 eno49: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[Thu Sep 16 01:43:52 2021] ixgbe 0000:06:00.0 eno49: NIC Link is Down
[Thu Sep 16 01:43:52 2021] bond0: (slave eno49): link status definitely down, disabling slave
[Thu Sep 16 01:43:52 2021] bond0: (slave eno50): making interface the new active one
[Thu Sep 16 01:43:52 2021] device eno49 left promiscuous mode
[Thu Sep 16 01:43:52 2021] device eno50 entered promiscuous mode
[Thu Sep 16 01:43:53 2021] ixgbe 0000:06:00.0 eno49: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[Thu Sep 16 01:43:53 2021] bond0: (slave eno49): link status definitely up, 10000 Mbps full duplex
[Thu Sep 16 01:43:53 2021] bond0: (slave eno49): making interface the new active one
[Thu Sep 16 01:43:53 2021] device eno50 left promiscuous mode
[Thu Sep 16 01:43:53 2021] device eno49 entered promiscuous mode
[Thu Sep 16 01:51:07 2021] ixgbe 0000:06:00.0 eno49: initiating reset due to tx timeout
[Thu Sep 16 01:51:07 2021] ixgbe 0000:06:00.0 eno49: Reset adapter
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jul 23 06:37 seq
 crw-rw---- 1 root audio 116, 33 Jul 23 06:37 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.20
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2021-07-19 (58 days ago)
InstallationMedia: Ubuntu-Server 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant BL460c Gen9
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-42-generic root=/dev/mapper/vg0-root ro spectre_v2=off nopti pti=off nospec_store_bypass_disable spec_store_bypass_disable=off l1tf=off hugepagesz=2M hugepages=30000 transparent_hugepage=never
ProcVersionSignature: Ubuntu 5.4.0-42.46-generic 5.4.44
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-42-generic N/A
 linux-backports-modules-5.4.0-42-generic N/A
 linux-firmware 1.187.2
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.4.0-42-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/17/2017
dmi.bios.vendor: HP
dmi.bios.version: I36
dmi.chassis.type: 28
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrI36:bd02/17/2017:svnHP:pnProLiantBL460cGen9:pvr:cvnHP:ct28:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant BL460c Gen9
dmi.product.sku: 727021-B21
dmi.sys.vendor: HP

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1943879

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: focal
Revision history for this message
Satish Patel (satish-txt) wrote : CRDA.txt

apport information

tags: added: apport-collected uec-images
description: updated
Revision history for this message
Satish Patel (satish-txt) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : Lspci.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : Lspci-vt.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : Lsusb.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : Lsusb-t.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : Lsusb-v.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : ProcModules.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : UdevDb.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : WifiSyslog.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote : acpidump.txt

apport information

Revision history for this message
Satish Patel (satish-txt) wrote :

Find attached Report

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Satish Patel (satish-txt) wrote :

Any update here?

Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

Please test latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.2/amd64/

Headers are not needed.

Changed in linux (Ubuntu):
status: Expired → Confirmed
Revision history for this message
can (canzhu) wrote :

The same with this? and any updates there?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.