linux-image-3.13.0-24-generic kernel doesn't boot on Xen 3.0 from CentOS5

[Ed Szynaka]

linux-image-3.13.0-24-generic does not boot on Xen 3.0. Tested on CentOS 5.10. Boot crashes before any output from kernel.

Output from xm dmesg:
(XEN) event_channel.c:181:d0 EVTCHNOP failure: domain 0, error -22, line 181
(XEN) event_channel.c:181:d0 EVTCHNOP failure: domain 0, error -22, line 181
(XEN) traps.c:1912:d2004 Domain attempted WRMSR 000000000000008b from 00000015:00000000 to 00000000:00000000.
(XEN) traps.c:405:d2004 Unhandled invalid opcode fault/trap [#6] in domain 2004 on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 2004 (vcpu#0) crashed on cpu#14:
(XEN) ----[ Xen-3.1.2-308.8.2.el5 x86_64 debug=n Not tainted ]----
(XEN) CPU: 14
(XEN) RIP: e033:[<ffffffff81004c38>]
(XEN) RFLAGS: 0000000000000282 CONTEXT: guest
(XEN) rax: ffffffffffffffea rbx: 00000000023ff000 rcx: 0000000000000002
(XEN) rdx: 0000000000000001 rsi: ffffffff81c01e48 rdi: 0000000000000000
(XEN) rbp: ffffffff81c01e58 rsp: ffffffff81c01e08 r8: 0000000000000000
(XEN) r9: ffffffff81c01dc8 r10: 0000000000007ff0 r11: 00000000ffffffff
(XEN) r12: ffffffff81dd6900 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000000026b0
(XEN) cr3: 0000001dde584000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffffffff81c01e08:
(XEN) 0000000000000002 00000000ffffffff ffffffff81004c38 000000010000e030
(XEN) 0000000000010082 ffffffff81c01e48 000000000000e02b ffffffff81004c32
(XEN) ffffffff00010004 ffffffff8171e4a0 ffffffff81c01e78 ffffffff81d38f1d
(XEN) ffffffff00010001 ffffffff81727ed0 ffffffff81c01ef0 ffffffff81d3c58b
(XEN) 0000000000000000 0000000000000000 ffffffff81c01ef0 ffffffff81c01f08
(XEN) ffffffff00000010 ffffffff81c01f00 ffffffff81c01ec0 ffffffff81ebc632
(XEN) ffffffffffffffff ffffffff81dd6900 0000000000000000 0000000000000000
(XEN) 0000000000000000 ffffffff81c01f30 ffffffff81d34c07 0000000000000018
(XEN) 356a4e4bd08da87b ffffffff81c01f60 00000000f3dfffff 0000000000000000
(XEN) 0000000000000000 ffffffff81c01f40 ffffffff81d345ee ffffffff81c01ff8
(XEN) ffffffff81d384a7 000206c200000000 0000000133200800 000000000f898371
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0f00000060c0c748
(XEN) ccccccccccccc305 cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN) cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN) cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN) cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc

Appears to be a known bug that has not been patched in mainline kernel yet:
http://lists.xen.org/archives/html/xen-devel/2014-05/msg01474.html

Tested recompiling from kernel at git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git with patch from above thread.

# lsb_release -rd
Description: Ubuntu 14.04 LTS
Release: 14.04

# apt-cache policy linux-image-3.13.0-24-generic
linux-image-3.13.0-24-generic:
  Installed: 3.13.0-24.46
  Candidate: 3.13.0-24.46
  Version table:
 *** 3.13.0-24.46 0
        100 /var/lib/dpkg/status
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 22 15:10 seq
 crw-rw---- 1 root audio 116, 33 May 22 15:10 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 20.010048] init: plymouth-upstart-bridge main process ended, respawning
DistroRelease: Ubuntu 14.04
IwConfig: Error: [Errno 2] No such file or directory
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
Package: linux (not installed)
PciMultimedia:

ProcFB:

ProcKernelCmdLine: root=/dev/xvda1 ro quiet splash
ProcModules:
 kvm 451511 1 kvm_intel, Live 0xffffffffa0000000
 kvm_intel 143060 0 - Live 0xffffffffa0248000
ProcVersionSignature: Ubuntu 3.13.0-27.50-generic 3.13.11
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-27-generic N/A
 linux-backports-modules-3.13.0-27-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-27-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1322211

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

apport information

[Ed Szynaka (y-launchp0d-7)]

Added requested apport information from patched system. Unpatched version does not boot so cannot collect information from unpatched version.

[Stefan Bader (smb)]

I see that this was agreed to be an upstream change that is considered stable material. The patch looks reasonable from the discussion, but I would like to see it at least reaching upstream linux or linux-next at least.

[Tore Anderson (toreanderson)]

Hi, I'm also affected by this issue. I have a number of virtual machines running Precise which are now left in limbo - I cannot use the original Precise kernel (3.2.0) because of some missing features, and I cannot upgrade to the Trusty kernel (or upgrade the entire distribution to Trusty for that matter) due to this issue. I can run the Raring or Saucy kernels, but those are now End of Life, which means I won't get any security fixes. I have no influence over my VPS provider's choice of hypervisor software.

It's been three months since this bug saw any activity, has it been forgotten about?

[Stefan Bader (smb)]

Hm, sorry, unfortunately lost track of this bug report completely. The change to fix this was different but I think this is just the same "registering NMI" problem tracked in bug #1350375. That is now committed and should pulled in when the next kernel update is prepared.

[Tore Anderson (toreanderson)]

FYI, this bug is now fixed by http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-trusty.git;a=commit;h=6ac0d80c79b062b44135cec6436d6eeeaeed1ec2. I've tested linux-image-3.13.0-35-generic version 3.13.0-35.62~precise1 and can confirm it's working OK. So this bug report can probably be closed.

Tore

[Stefan Bader (smb)]

Closing according to last comment.