3.13: libvirtd: page allocation failure: order:4, mode:0x1040d0

[Dave Chiluk]

[Impact]
 * libvirtd is no longer able to open the vhost_net device. This causes the guest VM to hang. This happens if memory becomes fragmented to the point where vhost_net_open is not able to successfully kmalloc.

 * Gratuitous stack trace.
libvirtd: page allocation failure: order:4, mode:0x1040d0
CPU: 14 PID: 82768 Comm: libvirtd Not tainted 3.13.0-85-generic #129-Ubuntu
Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 1.5.4 10/002/2015
 0000000000000000 ffff88003b419990 ffffffff8172b6a7 00000000001040d0
 0000000000000000 ffff88003b419a18 ffffffff811580eb ffff88187fffce48
 ffff88003b4199b8 ffffffff8115abd6 ffff88003b4199e8 0000000000000286
Call Trace:
 [<ffffffff8172b6a7>] dump_stack+0x64/0x82
 [<ffffffff811580eb>] warn_alloc_failed+0xeb/0x140
 [<ffffffff8115abd6>] ? drain_local_pages+0x16/0x20
 [<ffffffff8115c8c0>] __alloc_pages_nodemask+0x980/0xb90
 [<ffffffff8119b3a3>] alloc_pages_current+0xa3/0x160
 [<ffffffff811570ae>] __get_free_pages+0xe/0x50
 [<ffffffff811743be>] kmalloc_order_trace+0x2e/0xc0
 [<ffffffffa04e79c9>] vhost_net_open+0x29/0x1b0 [vhost_net]
 [<ffffffff81484283>] misc_open+0xb3/0x170
 [<ffffffff811c63ff>] chrdev_open+0x9f/0x1d0
 [<ffffffff811bef13>] do_dentry_open+0x233/0x2e0
 [<ffffffff811c6360>] ? cdev_put+0x30/0x30
 [<ffffffff811bf249>] vfs_open+0x49/0x50
 [<ffffffff811d0812>] do_last+0x562/0x1370
 [<ffffffff811d16db>] path_openat+0xbb/0x670
 [<ffffffff811d2afa>] do_filp_open+0x3a/0x90
 [<ffffffff811df957>] ? __alloc_fd+0xa7/0x130
 [<ffffffff811c0d69>] do_sys_open+0x129/0x2a0
 [<ffffffff811c0efe>] SyS_open+0x1e/0x20
 [<ffffffff8173c39d>] system_call_fastpath+0x1a/0x1f

 * justification: because cloud.

 * The patches fix this issue by allowing vhost_net_open to use vmalloc when kmalloc fails to find a sufficient page size.

[Test Case]

 * Fragment Kernel memory. Write to Nic from within a kvm guest that uses a virtio nic.

[Regression Potential]

 * Fix was implemented upstream in 3.15, and still exists.

 * The fix is fairly straightfoward given the stack trace and the upstream fix.

 * The fix is hard to verify, as it requires significant memory fragmentation, and an over-active guest. The users machine that was experiencing this has worked around this by removing VM's from the compute host, and using vfs.cache.pressure=600.

[Other Info]

 * https://lkml.org/lkml/2013/1/23/492
 * http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=23cc5a991c7a9fb7e6d6550e65cee4f4173111c5
 * http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d04257b07f2362d4eb550952d5bf5f4241a8046d
 * I'm going on vacation, and Eric Desrochers will be following up on this in my absence. This is also the reason for submitting before receiving verification.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1616193

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dave Chiluk (chiluk)]

Shut the front door bug-bot.

[Dave Chiluk (chiluk)]

Fix submitted to <email address hidden>.

[Tim Gardner (timg-tpi)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Dave Chiluk (chiluk)]

Marking verification-done-trusty, as the user has not replied back to me one way or the other so I'll assume their kernel is no longer getting allocation errors. I definitely don't want this pulled from the sources prematurely.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-96.143

---------------
linux (3.13.0-96.143) trusty; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1618083

  * CVE-2015-8767
    - sctp: Prevent soft lockup when sctp_accept() is called during a timeout
      event

  * MacBookPro11,4 fails to poweroff or suspend (LP: #1587714)
    - SAUCE: PCI: Workaround to enable poweroff on Mac Pro 11

  * 3.13: libvirtd: page allocation failure: order:4, mode:0x1040d0
    (LP: #1616193)
    - vhost-net: extend device allocation to vmalloc
    - vhost-net: don't open-code kvfree

  * [arm64] nova instances can't boot with 3.13.0-92 (LP: #1608854)
    - Revert "UBUNTU: [Config] CONFIG_EFI=n for arm64"
    - Revert "UBUNTU: SAUCE: UEFI: Set EFI_SECURE_BOOT bit in x86_efi_facility"
    - Revert "UBUNTU: SAUCE: UEFI: Add secure boot and MOK SB State disabled
      sysctl"
    - Revert "UBUNTU: SAUCE: UEFI: Display MOKSBState when disabled"
    - Revert "UBUNTU: SAUCE: UEFI: efi: Disable secure boot if shim is in insecure
      mode"
    - Revert "UBUNTU: SAUCE: UEFI MODSIGN: Import certificates from UEFI Secure
      Boot"
    - Revert "UBUNTU: SAUCE: UEFI: efi: Make EFI_SECURE_BOOT_SIG_ENFORCE depend on
      EFI"
    - Revert "UBUNTU: SAUCE: UEFI: Add option to automatically enforce module
      signatures when in Secure Boot mode"
    - Revert "UBUNTU: [Config] CONFIG_EFI_SECURE_BOOT_SIG_ENFORCE=y"
    - Revert "UBUNTU: SAUCE: UEFI: x86: Restrict MSR access when module loading is
      restricted"
    - Revert "UBUNTU: SAUCE: UEFI: kexec: Disable at runtime if the kernel
      enforces module loading restrictions"
    - Revert "UBUNTU: SAUCE: UEFI: acpi: Ignore acpi_rsdp kernel parameter when
      module loading is restricted"
    - Revert "UBUNTU: SAUCE: UEFI: Restrict /dev/mem and /dev/kmem when module
      loading is restricted"
    - Revert "UBUNTU: SAUCE: UEFI: asus-wmi: Restrict debugfs interface when
      module loading is restricted"
    - Revert "UBUNTU: SAUCE: UEFI: ACPI: Limit access to custom_method"
    - Revert "UBUNTU: SAUCE: UEFI: x86: Lock down IO port access when module
      security is enabled"
    - Revert "UBUNTU: SAUCE: UEFI: PCI: Lock down BAR access when module security
      is enabled"
    - Revert "UBUNTU: SAUCE: UEFI: Add secure_modules() call"
    - Revert "x86/efi: Build our own EFI services pointer table"
    - Revert "efi: Add separate 32-bit/64-bit definitions"

  * [Hyper-V] storvsc messages for CD-ROM medium not present tray closed
    (LP: #1590655)
    - scsi: storvsc: Filter out storvsc messages CD-ROM medium not present

  * CVE-2016-3841
    - ipv6: add complete rcu protection around np->opt

 -- Kamal Mostafa <email address hidden> Tue, 16 Aug 2016 10:20:51 -0700