Xen HVM guests running linux 4.10 fail to boot on Intel hosts

[Stefan Bader]

Starting with Linux kernel 4.10, the kernel does some sanity checking on the TSC_ADJUST MSR. Xen has implemented some support for that MSR in the hypervisor (Xen 4.3 and later) for HVM guests. But boot and secondary vCPUs are set up inconsistently. This causes the boot of a 4.10 HVM guest to hang early on boot.

This was fixed in the hypervisor by:

  commit 98297f09bd07bb63407909aae1d309d8adeb572e
  x86/hvm: do not set msr_tsc_adjust on hvm_set_guest_tsc_fixed

That fix would be contained in 4.6.5 and 4.7.2 and would be in 4.8.1 (not released, yet) which mean that Ubuntu 14.04/16.04/16.10 and 17.04 currently are affected.

---

SRU Justification:

Impact: Without the TSC_ADJUST MSR fix, 4.10 and later kernels will get stuck at boot when running as (PV)HVM guests on Xen 4.3 and later.

Fix: Above fix either individually applied or as part of Xen stable stream (for Xen 4.7.x and 4.6.x) resolves the issue.

Testcase:
- Requires Intel based host which supports the TSC_ADJUST MSR
- Configured as Xen host
- HVM guest running Zesty/17.04
- Stuck at boot before, normal booting OS after

[Stefan Bader (smb)]

Adding linux task to show the relationship between the two. Basically starting with 4.10 the kernel verifies the feature and exposes the broken implementation.

[Stefan Bader (smb)]

Verified against prepared releases in: https://launchpad.net/~smb/+archive/ubuntu/xen

[Launchpad Janitor (janitor)]

This bug was fixed in the package xen - 4.8.0-1ubuntu2

---------------
xen (4.8.0-1ubuntu2) zesty; urgency=medium

  * Cherry-pick upstream change to fix TSC_ADJUST MSR handling in HVM
    guests running on Intel based hosts (LP: #1671760)

 -- Stefan Bader <email address hidden> Tue, 14 Mar 2017 09:27:04 +0100

[Brian Murray (brian-murray)]

Hello Stefan, or anyone else affected,

Accepted xen into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.7.2-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Stefan, or anyone else affected,

Accepted xen into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.6.5-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Stefan, or anyone else affected,

Accepted xen into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xen/4.4.2-0ubuntu0.14.04.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stefan Bader (smb)]

For Xenial, reinstalled Xen from proposed. Booted Zesty 4.10 HVM ok.

[Stefan Bader (smb)]

For Yakkety, reinstalled Xen from proposed. Booted Zesty 4.10 HVM ok.

[Stefan Bader (smb)]

For Xenial, reinstalled Xen from proposed. Booted Zesty 4.10 HVM ok.

[Stefan Bader (smb)]

... again. Had done that already last week...

[Stefan Bader (smb)]

Now successfully tested the Trusty proposed version.

[Launchpad Janitor (janitor)]

This bug was fixed in the package xen - 4.4.2-0ubuntu0.14.04.10

---------------
xen (4.4.2-0ubuntu0.14.04.10) trusty; urgency=medium

  * Backport upstream change to fix TSC_ADJUST MSR handling in HVM
    guests running on Intel based hosts (LP: #1671760)

 -- Stefan Bader <email address hidden> Tue, 14 Mar 2017 11:17:48 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for xen has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package xen - 4.6.5-0ubuntu1

---------------
xen (4.6.5-0ubuntu1) xenial; urgency=medium

  * Rebasing to upstream stable release 4.6.5 (LP: #1671864)
    https://www.xenproject.org/downloads/xen-archives/xen-46-series.html
    - Includes fix for booting 4.10 Linux kernels in HVM guests on Intel
      hosts which support the TSC_ADJUST MSR (LP: #1671760)
    - Additional security relevant changes:
      * CVE-2013-2076 / XSA-052 (update)
        - Information leak on XSAVE/XRSTOR capable AMD CPUs
      * CVE-2016-7093 / XSA-186 (4.6.3 became vulnerable)
        - x86: Mishandling of instruction pointer truncation during emulation
      * XSA-207
        - memory leak when destroying guest without PT devices
    - Replacing the following security fixes with the versions from the
      stable update:
      * CVE-2015-7812 / XSA-145
        - arm: Host crash when preempting a multicall
      * CVE-2015-7813 / XSA-146
        - arm: various unimplemented hypercalls log without rate limiting
      * CVE-2015-7814 / XSA-147
        - arm: Race between domain destruction and memory allocation decrease
      * CVE-2015-7835 / XSA-148
        - x86: Uncontrolled creation of large page mappings by PV guests
      * CVE-2015-7969 / XSA-149, XSA-151
        - leak of main per-domain vcpu pointer array
        - x86: leak of per-domain profiling-related vcpu pointer array
      * CVE-2015-7970 / XSA-150
        - x86: Long latency populate-on-demand operation is not preemptible
      * CVE-2015-7971 / XSA-152
        - x86: some pmu and profiling hypercalls log without rate limiting
      * CVE-2015-7972 / XSA-153
        - x86: populate-on-demand balloon size inaccuracy can crash guests
      * CVE-2016-2270 / XSA-154
        - x86: inconsistent cachability flags on guest mappings
      * CVE-2015-8550 / XSA-155
        - paravirtualized drivers incautious about shared memory contents
      * CVE-2015-5307, CVE-2015-8104 / XSA-156
        - x86: CPU lockup during exception delivery
      * CVE-2015-8338 / XSA-158
        - long running memory operations on ARM
      * CVE-2015-8339, CVE-2015-8340 / XSA-159
        XENMEM_exchange error handling issues
      * CVE-2015-8341 / XSA-160
        - libxl leak of pv kernel and initrd on error
      * CVE-2015-8555 / XSA-165
        - information leak in legacy x86 FPU/XMM initialization
      * XSA-166
        - ioreq handling possibly susceptible to multiple read issue
      * CVE-2016-1570 / XSA-167
        - PV superpage functionality missing sanity checks
      * CVE-2016-1571 / XSA-168
        - VMX: intercept issue with INVLPG on non-canonical address
      * CVE-2015-8615 / XSA-169
        - x86: unintentional logging upon guest changing callback method
      * CVE-2016-2271 / XSA-170
        - VMX: guest user mode may crash guest with non-canonical RIP
      * CVE-2016-3158, CVE-2016-3159 / XSA-172
        - broken AMD FPU FIP/FDP/FOP leak workaround
      * CVE-2016-3960 / XSA-173
        - x86 shadow pagetables: address width overflow
      * CVE-2016-4962 / XSA-175
        - Unsanitised guest input in libxl device handling code
      * CVE-2016-4480 / XSA-176
        - x86 ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package xen - 4.7.2-0ubuntu1

---------------
xen (4.7.2-0ubuntu1) yakkety; urgency=medium

  * Rebasing to upstream stable release 4.7.2 (LP: #1672767)
    https://www.xenproject.org/downloads/xen-archives/xen-47-series.html
    - Includes fix for booting 4.10 Linux kernels in HVM guests on Intel
      hosts which support the TSC_ADJUST MSR (LP: #1671760)
    - Dropping: d/p/preup-tools-fix-linear-p2m-save.patch which is part
      of the stable update.
    - Additional security relevant changes:
      * XSA-207
        - memory leak when destroying guest without PT devices
    - Replacing the following security fixes with the versions from the
      stable update:
      * CVE-2016-6258 / XSA-182
        - x86: Privilege escalation in PV guests
      * CVE-2016-6259 / XSA-183
        - x86: Missing SMAP whitelisting in 32-bit exception / event delivery
      * CVE-2016-7092 / XSA-185
        - x86: Disallow L3 recursive pagetable for 32-bit PV guests
      * CVE-2016-7093 / XSA-186
        - x86: Mishandling of instruction pointer truncation during emulation
      * CVE-2016-7094 / XSA-187
        - x86 HVM: Overflow of sh_ctxt->seg_reg[]
      * CVE-2016-7777 / XSA-190
        - CR0.TS and CR0.EM not always honored for x86 HVM guests
      * CVE-2016-9386 / XSA-191
        - x86 null segments not always treated as unusable
      * CVE-2016-9382 / XSA-192
        - x86 task switch to VM86 mode mis-handled
      * CVE-2016-9385 / XSA-193
        - x86 segment base write emulation lacking canonical address checks
      * CVE-2016-9384 / XSA-194
        - guest 32-bit ELF symbol table load leaking host data
      * CVE-2016-9383 / XSA-195
        - x86 64-bit bit test instruction emulation broken
      * CVE-2016-9377, CVE-2016-9378 / XSA-196
        - x86 software interrupt injection mis-handled
      * CVE-2016-9379, CVE-2016-9380 / XSA-198
        - delimiter injection vulnerabilities in pygrub
      * CVE-2016-9932 / XSA-200
        - x86 CMPXCHG8B emulation fails to ignore operand size override
      * CVE-2016-9815, CVE-2016-9816, CVE-2016-9817, CVE-2016-9818 / XSA-201
        - ARM guests may induce host asynchronous abort
      * CVE-2016-10024 / XSA-202
        - x86 PV guests may be able to mask interrupts
      * CVE-2016-10025 / XSA-203
        - x86: missing NULL pointer check in VMFUNC emulation
      * CVE-2016-10013 / XSA-204
        - x86: Mishandling of SYSCALL singlestep during emulation
  * Copy contents of debian/build/install-utils_$(ARCH)/usr/sbin into
    debian/build/install-utils_$ARCH/usr/lib/xen-$(VERSION) (LP: #1396670).

 -- Stefan Bader <email address hidden> Tue, 14 Mar 2017 15:45:59 +0100