HP Proliant Serverrs - DL360 and DL380 Gen8 - Precise Kernel Panic - General Protection Fault and X2APIC/XAPIC boot parameters

[Rafael David Tinoco]

It was brought to my attention the following situation:

"""
We massively upgraded our Ubuntu 12.04 servers (most of them are HP
DL360p Gen8 or DL380 Gen8) to 3.2.0-67 kernel And in the last 2-3
days we already had to reboot 5 of them because they completely hang

Some of them had the following messages under syslog :
kernel: [384707.675479] general protection fault: 0000 [#5666] SMP

others had :
kernel: [950725.612724] BUG: unable to handle kernel paging request

All of them have this also :
your BIOS is broken and requested that x2apic be disabled
"""

Comments bellow

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1398497

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Rafael David Tinoco (rafaeldtinoco)]

Just got confirmation from HP regarding X2APIC and PROLIANT SERVERS using Ubuntu Linux:

Before GEN8.. all Proliant Servers did NOT support X2APIC. They had firmware saying OS to NOT use X2APIC (and OS started supporting this by commit: 41750d3, already included in kernel 3.2). The thing is.. for these servers, opting out from X2APIC made them to use XAPIC IRQ remapping (not supported).

-----

So, for Proliant Servers BEFORE GEN8 the recommended cmdline is this:

"nox2apic intermap=off"

Obs: nox2apic might not be needed since firmware is saying for Linux to optout from using x2apic. Anyway I prefer to recommend this flag to make sure kernels before 3.2 still work (not the case here).

-----

AFTER GEN8 (inclusive), the firmware STILL says that X2APIC must not be used, but they ARE indeed supported by GEN8 (DL360, DL380). So proper cmdlines are:

"intremap=no_x2apic_optout" # let X2APIC enabled with IRQ remapping

OR

"nox2apic intermap=off" # disable X2APIC AND IRQ remapping

X2APIC for these machines ONLY differs from XAPIC in question of IRQ remapping (easier to implement). The other difference, where x2apic is capable of addressing more CPUs, is not needed cause # of CPUs is low enough.

This points out to my last finding, regarding:

*** cdcd629869fabcd38ebd24a03b0a05ec1cbcafb0 x86: Fix and improve cmpxchg_double{,_local}()
|__> !!!!!!!!!!!!!!!! fix several problems related to cmpxchg and 64bits !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

cmpxchg is where instruction pointer is at (looks like) on the analyzed dump.

Problem here is that kernel version from the core was erased from ddebs.ubuntu and I'm using the "next" ddeb version. So I'm saying this is the "most likely" to be happening since I can't objdump binary used by analyzed system.

Things to be done:

1) Provide hotfix with this fix: cdcd629869f
2) Wait for intel_idle problem observation (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1318551)
3) Recommend users either:

"intremap=no_x2apic_optout" OR "nox2apic intermap=off"

TOGETHER with "intel_idle.max_cstate=0"

until we fix intel_idle.

Thank you

Rafael Tinoco

[Esel (glumpad)]

ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] Datei oder Verzeichnis nicht gefunden
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=UUID=30721826-134d-4032-8d87-9213511577b5
InstallationDate: Installed on 2014-11-07 (34 days ago)
InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3)
MachineType: HP ProLiant ML330 G6
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=de_AT:de
 TERM=xterm
 PATH=(custom, no user)
 LANG=de_AT.UTF-8
 SHELL=/bin/bash
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-39-generic root=UUID=ac0b8a1b-5d1a-4043-98c2-82819f497f51 ro nomdmonddf nomdmonisw
ProcVersionSignature: Ubuntu 3.13.0-39.66-generic 3.13.11.8
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-39-generic N/A
 linux-backports-modules-3.13.0-39-generic N/A
 linux-firmware 1.127.10
RfKill:

Tags: trusty trusty
Uname: Linux 3.13.0-39-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 07/02/2013
dmi.bios.vendor: HP
dmi.bios.version: W07
dmi.chassis.type: 7
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrW07:bd07/02/2013:svnHP:pnProLiantML330G6:pvr:cvnHP:ct7:cvr:
dmi.product.name: ProLiant ML330 G6
dmi.sys.vendor: HP

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

apport information

[Esel (glumpad)]

In my case it's not happening when I boot the Server but when it's already running. From time to time it freezes with the NMI-Error and I can't do anything except hard power-off and power-on. We're running virtualbox on it with 2 guests.

[Rafael David Tinoco (rafaeldtinoco)]

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1318551/comments/31

and beyond.

[Esel (glumpad)]

Thanks for your comment. Added only the "nox2apic intermap=off" and the error occoured again. Now added also the intel command, so the cmdline looks like "nox2apic intermap=off intel_idle.max_cstate=0". Is this right for my DL360 Gen6?

[Esel (glumpad)]

Hi Rafael,

sry but this is not my server, its one of a customer who has an important program running on it, so I'm sorry but I can't test around. I'm now running the right cmdline (entered in /etc/default/grub under GRUB_CMDLINE_LINUX=).
Because the edit needed a restart i installed linux-crashdump before so if the error throws up again I'd like to accept your offer to have a look at it! FYI: I havn't had to do a lot with things this deep in kernel, so keywords like "linux-crashdump" helps me to find the right thing :).

Greetings from Austria

[Dave (xwebsubs)]

Hello,

Sorry if my posting is not appropriate... I'm very new to Linux, but inquisitive.
I have just installed an Ubuntu 14.04 based desktop, Linux Lite 2.2 kernel 3.13.?
H/W is Intel CPU i7-4790S = 4 Core +HT and I have VT enabled so I can run Virtualbox.

I have found the following in dmesg log file:

[ 0.022633] Your BIOS is broken and requested that x2apic be disabled.
[ 0.022633] This will slightly decrease performance.
[ 0.022633] Use 'intremap=no_x2apic_optout' to override BIOS request.
[ 0.022745] Enabled IRQ remapping in xapic mode
[ 0.022746] x2apic not enabled, IRQ remapping is in xapic mode

Is this also related to the issue in this thread.?
If it it is,
I can provide more information if you tell me what I need to do.?

David

[Esel (glumpad)]

So with the correct cmdline the system is now running since 6 days. Will report if somethings happening.

[Rafael David Tinoco (rafaeldtinoco)]

Dave,

This message is being given to let you know that your BIOS (EFI/firmware/..) is "asking" the OS to NOT USE x2apic (and this happens on HP Proliant Servers). This is because, before Gen7, Proliant servers did not support X2APIC (a new Programmable Interrupt Controller) and had to operate into "XAPIC"/"APIC" compatible mode (even if their CPU supported X2APIC).

For Proliant Servers >= Gen8 you can say to Linux to "ignore" this request from firmware. To do that you change

GRUB_CMDLINE_LINUX_DEFAULT="..." to
GRUB_CMDLINE_LINUX_DEFAULT="... intremap=no_x2apic_optout"

inside /etc/default/grub and run "update-grub" command (rebooting your server later on).

This is because Gen8 are X2APIC capable but the firmware still asks OS not to enable it (as I was told by HP).

If you use HP Proliant Servers < Gen8 you have to use:

"nox2apic intermap=off"

For not having problems regarding IRQ handling, deactivating x2apic.

Note that if you are using HP Proliant Servers you have to be aware of the following bug also:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1318551

adding "intel_idle.max_cstate=0" to the cmdline also.

[Dave (xwebsubs)]

Hello Rafael,

Many thanks, O.K
So I can ignore this if I am "not" using Proliant Server.?

Dave

[Esel (glumpad)]

Hi Rafael,

error happened again after about 3 weeks without kernel panic. Can you please tell me how to get the crashdump out of the machine so I can Post it here?

Thanks!

[Esel (glumpad)]

or do you have a Link where I can look? Had a look at crashdump but I'm not familiar with and don't know what you neeed...

[Rafael David Tinoco (rafaeldtinoco)]

Esel,

Could you please follow instructions from:

https://sites.google.com/site/inaddyorg/texts/installing-ubuntu-crash-dump

Taking the opportunity to leave comments if you find anything wrong in the procedure ?

Tks, looking forward to receiving core from you.

Tks

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu Precise) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]