INFO: task bash:15633 blocked for more than 120 seconds

[Tim Stockford]

Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Oct 11 04:55 seq
 crw-rw---- 1 root audio 116, 33 Oct 11 04:55 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.12
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2015-08-20 (51 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
MachineType: HP ProLiant DL380 Gen9
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.16.0-48-generic root=UUID=3a22e2b7-154a-41ed-8b5c-0c02193172bd ro
ProcVersionSignature: Ubuntu 3.16.0-48.64~14.04.1-generic 3.16.7-ckt15
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-48-generic N/A
 linux-backports-modules-3.16.0-48-generic N/A
 linux-firmware 1.127.15
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.16.0-48-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 05/06/2015
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP89:bd05/06/2015:svnHP:pnProLiantDL380Gen9:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL380 Gen9
dmi.sys.vendor: HP

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1504801

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

apport information

[Tim Stockford (tim-stockford)]

I was unable to send 'apport-collect' prior to a reboot. I let '*** Collecting problem information' run for an hour before quitting out and scheduling a reboot.

The only error reported in kernel log since the reboot is below:

Oct 11 04:55:17 ustapp01n kernel: [ 1.343088] ioapic: probe of 0000:00:05.4 failed with error -22
Oct 11 04:55:17 ustapp01n kernel: [ 1.343100] ioapic: probe of 0000:80:05.4 failed with error -22
Oct 11 04:55:17 ustapp01n kernel: [ 1.413867] ERST: Error Record Serialization Table (ERST) support is initialized.

Thanks,

Tim

[penalvch (penalvch)]

Tim Stockford, to confirm:
1) Was the message you are reporting about (INFO: task bash:15633 blocked for more than 120 seconds) spewed out to the console?
2) Were or are there any crash files in /var/crash?

[Tim Stockford (tim-stockford)]

Console messages are unconfirmed and there is no /var/crash.

However, the end user is reporting sshfs to be in use around the time of the hang. Due to what appears to be an issue in sshfs. I cannot confirm the exact timings match.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776849

I do see RX dropped packets incrementing on this host (and the partner host) since the most recent reboot.

I am wondering if sshfs and NIC drops/issues have resulted in a the Hung task messages. What do you guys think?

Thanks,

Tim

[penalvch (penalvch)]

Tim Stockford, I'll re-add the linux task for now, as a NIC driver issue, causing an SSHFS issue, causing a bash timeout issue would want to be investigated also, versus pure user space issue.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Tim Stockford (tim-stockford)]

Hi,

We are seeing an increasing amount of dropped RX packets on this and the partner host. We are running a bonded configuration but I removed the bonding and tested with single interfaces to see if the RX drops still increased, of which they did.

We have a mixture of NIC's

Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe

All seem to increase in dropped packets, the host is idle.

Using the Broadcom BCM5719 as an example. Over a period of ten minutes the interface received 2.87kb and dropped 3 additional RX packets.

Other than the original hung task we have not seen any further issues.

We are moving away from sshfs in this setup but are keen to understand the reason for the dropped RX packets.

Thanks,

Tim

[penalvch (penalvch)]

Tim Stockford, as per http://h20564.www2.hpe.com/hpsc/swd/public/readIndex?sp4ts.oid=7271242&swLangOid=8&swEnvOid=4186 an update to your computer's buggy and outdated BIOS is available (1.50). If you update to this following https://help.ubuntu.com/community/BIOSUpdate does it change anything?

If it doesn't, could you please both specify what happened, and provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

For more on BIOS updates and linux, please see https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette .

Please note your current BIOS is already in the Bug Description, so posting this on the old BIOS would not be helpful. As well, you don't have to create a new bug report.

Once the BIOS is updated, if the problem is still reproducible, and the information above is provided, then please mark this report Status Confirmed. Otherwise, please mark this as Invalid.

Thank you for your understanding.

[Tim Stockford (tim-stockford)]

The BIOS was updated:

P89
05/06/2015

We are still seeing drops on the bonded interfaces and the slave nic. The host has not been rebooted since the BIOS upgrade so all drop counts are new and current.

For example:

auto em4
iface em4 inet manual
 bond-master bond3
 bond-primary em4

auto em2
iface em2 inet manual
        bond-master bond3

bond3 - RX dropped:2581
em2 - RX dropped:1290
em4 - RX dropped:0

During a previous test (post reboot) I broke the bonding as still saw drops on em4 with minimal data. See my comment from 2015-10-21

Happy to provide further details upon request.

[penalvch (penalvch)]

Tim Stockford, to clarify, you ran the installer version 1.50_07-20-2015?