[Hyper-V] Ubuntu VM crash during Hyper-V backup or live migration after installing kernel 4.4.0-72

[Andrey Vertexx]

Description: Ubuntu 14.04.5 LTS
Release: 14.04

Hi, after installing kernel 4.4.0-67 or later I cannot backup my Ubuntu VM's on Hyper-V.
Microsoft Hyper-v 2012r2 Gen2 VMs

See Attachment for what happens is immediately after backup starts I get an error.

Eventually the kernel reports it has run out of memory and the machine just continuously scrolls errors message related to page allocation.

When reseting the virtual machine no logs can be found of the problem.

kernel 4.4.0-72-generic problem still here

DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2016-02-20 (409 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
Package: linux-image-4.4.0-72-generic 4.4.0-72.93~14.04.1
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=ru_RU.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49
SourcePackage: linux-lts-xenial
Tags: trusty
Uname: Linux 4.4.0-72-generic x86_64
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 5 12:05 seq
 crw-rw---- 1 root audio 116, 33 Apr 5 12:05 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.23
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
InstallationDate: Installed on 2016-02-20 (409 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=C
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro
ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-72-generic N/A
 linux-backports-modules-4.4.0-72-generic N/A
 linux-firmware 1.127.23
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 4.4.0-72-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 5894-4187-8369-8212-0547-2747-15
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 10 10:33 seq
 crw-rw---- 1 root audio 116, 33 Apr 10 10:33 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.23
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
InstallationDate: Installed on 2016-02-20 (414 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=C
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-73-generic root=/dev/mapper/ubuntu--vg-root ro
ProcVersionSignature: Ubuntu 4.4.0-73.94~lp1679898v2-generic 4.4.59
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-73-generic N/A
 linux-backports-modules-4.4.0-73-generic N/A
 linux-firmware 1.127.23
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty trusty
Uname: Linux 4.4.0-73-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 5894-4187-8369-8212-0547-2747-15
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 26 09:58 seq
 crw-rw---- 1 root audio 116, 33 Apr 26 09:58 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.23
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
InstallationDate: Installed on 2016-02-20 (430 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=C
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-75-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro
ProcVersionSignature: Ubuntu 4.4.0-75.96~14.04.1-generic 4.4.59
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-75-generic N/A
 linux-backports-modules-4.4.0-75-generic N/A
 linux-firmware 1.127.23
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty trusty
Uname: Linux 4.4.0-75-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 5894-4187-8369-8212-0547-2747-15
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 17 20:37 seq
 crw-rw---- 1 root audio 116, 33 May 17 20:37 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.23
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
 [ 28.480339] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
 [ 28.528215] ip6_tables: (C) 2000-2006 Netfilter Core Team
 [ 30.122225] init: plymouth-upstart-bridge main process ended, respawning
 [ 53.463858] hv_balloon: Received INFO_TYPE_MAX_PAGE_CNT
 [ 53.463891] hv_balloon: Data Size is 8
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
InstallationDate: Installed on 2016-02-20 (452 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=C
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-78-generic.efi.signed root=/dev/mapper/ubuntu--vg-root ro
ProcVersionSignature: Ubuntu 4.4.0-78.99~14.04.2-generic 4.4.62
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-78-generic N/A
 linux-backports-modules-4.4.0-78-generic N/A
 linux-firmware 1.127.23
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty trusty
Uname: Linux 4.4.0-78-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 5894-4187-8369-8212-0547-2747-15
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1679898

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Andrey Vertexx (vertexx)]

Seconв screenshot

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

I use System Center Data Protection Manager 2012 R2 UR12 for a backup. DPM use VSS for a virtual machine online backup. Also this problem happens during live migration Ubuntu VM from one Hyper-V node to another.
Rolled back to kernel 4.4.0-66-generic solve the problem. It is very crytical issue for our company.

[Joseph Salisbury (jsalisbury)]

Can you test the upstream 4.4.49 kernel? That will tell us if this is specific to an Ubuntu change, or if the bug is also in upstream stable.

The 4.4.49 kernel can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.49/

[Joseph Salisbury (jsalisbury)]

This may be a duplicate of bug 1677938

[Joseph Salisbury (jsalisbury)]

Do you have an easy way to reproduce the bug? If so, I'd like to also give you some test kernels to try.

The only two hyper-v specific commits between -66 and -67 are the following:

c96360201be2 UBUNTU: SAUCE: pci-hyperv: lock pci bus on device eject
909a8ec8b1ca UBUNTU: SAUCE: pci-hyperv: properly handle pci bus remove

[Joseph Salisbury (jsalisbury)]

Actually, there is a third hyper-v change between -66 and -67 as well:
ecc705cb8622 UBUNTU: SAUCE: PCI: hv: Fix wslot_to_devfn() to fix warnings on device removal

[Joseph Salisbury (jsalisbury)]

All three of these commits came in via bug 1665097

[Joseph Salisbury (jsalisbury)]

Three hyper-v commits also came in via bug 1661430:

dda4a1d86bc4 Drivers: hv: vmbus: finally fix hv_need_to_signal_on_read()
eefabd3d3d65 Drivers: hv: vmbus: On the read path cleanup the logic to interrupt the host
2560f3f3bc65 Drivers: hv: vmbus: On write cleanup the logic to interrupt the host

[Joshua R. Poulson (jrp)]

I think this issue is with this set of storvsc changes in .67:

  * [Hyper-V] Bug fixes for storvsc (tagged queuing, error conditions)
    (LP: #1663687)
    - scsi: storvsc: Enable tracking of queue depth
    - scsi: storvsc: Remove the restriction on max segment size
    - scsi: storvsc: Enable multi-queue support
    - scsi: storvsc: use tagged SRB requests if supported by the device
    - scsi: storvsc: properly handle SRB_ERROR when sense message is present
    - scsi: storvsc: properly set residual data length on errors

We know of a regression with the last patch in this list and it is fixed with this upstream commit:

commit f1c635b439a5c01776fe3a25b1e2dc546ea82e6f
Author: Stephen Hemminger <email address hidden>
Date: Tue Mar 7 09:15:53 2017 -0800

    scsi: storvsc: Workaround for virtual DVD SCSI version

    Hyper-V host emulation of SCSI for virtual DVD device reports SCSI
    version 0 (UNKNOWN) but is still capable of supporting REPORTLUN.

    Without this patch, a GEN2 Linux guest on Hyper-V will not boot 4.11
    successfully with virtual DVD ROM device. What happens is that the SCSI
    scan process falls back to doing sequential probing by INQUIRY. But the
    storvsc driver has a previous workaround that masks/blocks all errors
    reports from INQUIRY (or MODE_SENSE) commands. This workaround causes
    the scan to then populate a full set of bogus LUN's on the target and
    then sends kernel spinning off into a death spiral doing block reads on
    the non-existent LUNs.

    By setting the correct blacklist flags, the target with the DVD device
    is scanned with REPORTLUN and that works correctly.

    Patch needs to go in current 4.11, it is safe but not necessary in older
    kernels.

    Signed-off-by: Stephen Hemminger <email address hidden>
    Reviewed-by: K. Y. Srinivasan <email address hidden>
    Reviewed-by: Christoph Hellwig <email address hidden>
    Signed-off-by: Martin K. Petersen <email address hidden>

[Joseph Salisbury (jsalisbury)]

I built a test kernel from the Xenial master-next branch, which includes the following commit:

f1c635b439a5c01776fe3a25b1e2dc546ea82e6f
Author: Stephen Hemminger <email address hidden>
Date: Tue Mar 7 09:15:53 2017 -0800

    scsi: storvsc: Workaround for virtual DVD SCSI version

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1679898/

Can you test this kernel and see if it resolves this bug?

[Andrey Vertexx (vertexx)]

A can test the kernel, but i need a full command list to install it - i am not linux prof.)

[Andrey Vertexx (vertexx)]

I have a one Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-71-generic x86_64) VM with the same problem.

[Andrey Vertexx (vertexx)]

I tried to install kernel.

/home/master/kerneltest01

wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-cloud-tools-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-cloud-tools-4.4.0-72_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-cloud-tools-common_4.4.0-72.93~lp1679898_all.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-doc_4.4.0-72.93~lp1679898_all.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-headers-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-headers-4.4.0-72_4.4.0-72.93~lp1679898_all.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-image-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-image-extra-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-source-4.4.0_4.4.0-72.93~lp1679898_all.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-tools-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-tools-4.4.0-72_4.4.0-72.93~lp1679898_amd64.deb
wget http://kernel.ubuntu.com/~jsalisbury/lp1679898/linux-tools-common_4.4.0-72.93~lp1679898_all.deb

sudo dpkg -i linux-*.deb

root@hq-srv-web-02:/home/master/kerneltest01# sudo dpkg -i linux-*.deb
Выбор ранее не выбранного пакета linux-cloud-tools-4.4.0-72.
(Чтение базы данных … на данный момент установлено 103349 файлов и каталогов.)
Подготовка к распаковке linux-cloud-tools-4.4.0-72_4.4.0-72.93~lp1679898_amd64.deb …
Распаковывается linux-cloud-tools-4.4.0-72 (4.4.0-72.93~lp1679898) …
Подготовка к распаковке linux-cloud-tools-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb …
Распаковывается linux-cloud-tools-4.4.0-72-generic (4.4.0-72.93~lp1679898) на замену (4.4.0-72.93~14.04.1) …
Подготовка к распаковке linux-cloud-tools-common_4.4.0-72.93~lp1679898_all.deb …
Распаковывается linux-cloud-tools-common (4.4.0-72.93~lp1679898) на замену (3.13.0-116.163) …
Выбор ранее не выбранного пакета linux-doc.
Подготовка к распаковке linux-doc_4.4.0-72.93~lp1679898_all.deb …
Распаковывается linux-doc (4.4.0-72.93~lp1679898) …
Подготовка к распаковке linux-headers-4.4.0-72_4.4.0-72.93~lp1679898_all.deb …
Распаковывается linux-headers-4.4.0-72 (4.4.0-72.93~lp1679898) на замену (4.4.0-72.93~14.04.1) …
Подготовка к распаковке linux-headers-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb …
Распаковывается linux-headers-4.4.0-72-generic (4.4.0-72.93~lp1679898) на замену (4.4.0-72.93~14.04.1) …
Подготовка к распаковке linux-image-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb …
Done.
Распаковывается linux-image-4.4.0-72-generic (4.4.0-72.93~lp1679898) на замену (4.4.0-72.93~14.04.1) …
dpkg: ошибка при обработке архива linux-image-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb (--install):
 попытка перезаписать «/lib/modules/4.4.0-72-generic/kernel/fs/aufs/aufs.ko», который уже имеется в пакете linux-image-extra-4.4.0-72-generic 4.4.0-72.93~14.04.1
dpkg-deb: ошибка: подпроцесс вставка уничтожен по сигналу (Обрыв канала)
Exam...

[Andrey Vertexx (vertexx)]

I can't install the kernel...

[Joseph Salisbury (jsalisbury)]

You would only need to install the linux-image and linux-image-extra .deb packages.

[Andrey Vertexx (vertexx)]

I have installed the linux-image-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb and linux-image-extra-4.4.0-72-generic_4.4.0-72.93~lp1679898_amd64.deb

After that the system won't boot.

[Andrey Vertexx (vertexx)]

install log

[Andrey Vertexx (vertexx)]

boot screenshot

[Andrey Vertexx (vertexx)]

second boot screenshot

[Joseph Salisbury (jsalisbury)]

I built a v2 version of the test kernel.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1679898/

Can you test this kernel and see if it resolves this bug?

[Steven Delombaerde (sdelombaerde)]

Hello

I have this problem too. I am running kernel 4.4.0-72-generic with up to date packages.
The symptoms are exactly as Deon described in the attached document.

This is very critical as block level backup on Hyper-V seems to be impossible, also disrupting backup of all other Hyper-V machines like Windows ...

Please advise.
Steven

[Joseph Salisbury (jsalisbury)]

@Steven, can you test the kernel posted in comment #30?

[Andrey Vertexx (vertexx)]

Today i have installed the new kernel and tools.

[Andrey Vertexx (vertexx)]

After reboot i have installed tools.

[Andrey Vertexx (vertexx)]

Linux hq-srv-web-02 4.4.0-73-generic #94~lp1679898v2 SMP Fri Apr 7 08:09:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@hq-srv-web-02:~# ps -ef | egrep "hv.*daemon"
root 1018 1 0 10:30 ? 00:00:00 /usr/lib/linux-tools/4.4.0-73-generic/hv_vss_daemon
root 1026 1 0 10:30 ? 00:00:00 /usr/lib/linux-tools/4.4.0-73-generic/hv_kvp_daemon
root 1996 1912 0 10:32 pts/2 00:00:00 egrep --color=auto hv.*daemon

root@hq-srv-web-02:~# ps -eaf|grep hv
root 123 2 0 10:30 ? 00:00:00 [hv_vmbus_con]
root 424 2 0 10:30 ? 00:00:00 [hv_balloon]
root 1018 1 0 10:30 ? 00:00:00 /usr/lib/linux-tools/4.4.0-73-generic/hv_vss_daemon
root 1026 1 0 10:30 ? 00:00:00 /usr/lib/linux-tools/4.4.0-73-generic/hv_kvp_daemon
root 1998 1912 0 10:33 pts/2 00:00:00 grep --color=auto hv

[Andrey Vertexx (vertexx)]

After that i tried to backup and live migration the VM.
And everythig works fine on first look.
Except some masseges on console screen during backup and live migration.

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Andrey Vertexx (vertexx)]

apport information

[Ian Gordon (ian-gordon)]

I can confirm that 4.4.0-66 does NOT crash and that 4.4.0-67 does indeed crash with out of memory errors.

I think I should say that I tried doing a VSS backup with 4.4.0-66 5 times and it didn't crash on any of them but it did log worrying messages (see attachment) which are very similar to what 4.4.0-67 gives.

[Ian Gordon (ian-gordon)]

test kernel 4.4.0-67.88~lp1679898Commitdf0a1b9cce531 works. Tested it 5 times and it does not run out of memory but it does temporarily detect 7 new disks (where as 4.4.0-66 only detected 3 new disks)

In the original 4.4.0-67 the extra/fake disks are detected and the extra /dev/sd* device files are still there after the end of the backup. Whereas the extra /dev/sd* files are not there after the backup in 4.4.0-66 and 4.4.0-67.88~lp1679898Commitdf0a1b9cce531

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
b06640ee040e347f122d0e885383da3efdb15ccd

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1679898/b06640ee040e347f122d0e885383da3efdb15ccd

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Ian Gordon (ian-gordon)]

Kernel 4.4.0-67.88~lp1679898Commitb06640ee040e34 (Comment #119) is NOT running out of memory.

[Andrey Vertexx (vertexx)]

I have NO probles with VM migration or backup with 4.4.0-83 kernel.

[Joseph Salisbury (jsalisbury)]

Great, thanks for the update Andrey. Can other also affected by this bug test the 4.4.0-83 kernel?

[Werner Reinersmann (belgarethde)]

I have updated three virtual machines to 4.4.0-83, two of them have attached physical disks.
The machines with physical drives will not complete the startup process (example included).
The machine without physical drives boots fine.

[Dave (furiousd)]

Unfortnately some of my machines with 4.4.0-83 are still freezing up when they do a hyper-v backup. I will attach the syslog.

[Dave (furiousd)]

Second server log

[Dave (furiousd)]

On the console screen they log out the following before they completely lock up.
sd 0:0:1:1 [sdc] Asking for cache data failed
sd 0:0:1:1 [sdc] Assuming drive cache: write through
Dev sdc: unable to read RDB block 0
Dev sdc unable to read RDB block 0
Dev sdc unable to read RDB block 0
INFO: task worker/1:255:77500 blocked for more than 120 seconds
 Not tainted 4.4.0-83-generic #106-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
(( Repeats several times ))
Out of Memory: Kill process 98158 (puppet) total-vm: 32100kB, anon-rss:0kB, file-rss:2260kB
blk_update_request: critical target error, dev sdd, sector 24
Buffer I/O error on dev sdd, logical block 3, async page read.

[randomname (randomname457)]

Hi,

We still experience this bug. We use Hyper-V 2012 and use Veeam to create VM backups (though this fact might be irrelevant). We upgraded our Ubuntu 14.04 to the latest xenial kernel (4.4.0-91) and still experience the VM hanging/freezing/stacktracing regularly directly after backup. Sometimes the server becomes unresponsive and shows the following error in the syslog: "Hyper-V VSS: VSS: freeze of /boot: Operation not supported" (after that it dies). The stacktrace included now did not crash the server, but it was definitely caused by the VM backup.

Some more info:
uname -a:

Linux SERVERNAME 4.4.0-91-generic #114~14.04.1-Ubuntu SMP Wed Aug 9 08:47:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

lsmod | grep 'hv_' :

hv_utils 28672 2
hv_storvsc 20480 260
ptp 20480 1 hv_utils
scsi_transport_fc 65536 1 hv_storvsc
hv_netvsc 40960 0
hv_vmbus 90112 6 hyperv_keyboard,hv_netvsc,hid_hyperv,hv_utils,hyperv_fb,hv_storvsc

dpkg --list | grep '4.4':

ii linux-cloud-tools-4.4.0-91-generic 4.4.0-91.114~14.04.1 amd64 Linux kernel version specific cloud tools for version 4.4.0-91
ii linux-cloud-tools-virtual-lts-xenial 4.4.0.91.75 amd64 This package will always depend on the latest minimal generic kernel cloud tools.
ii linux-generic-lts-xenial 4.4.0.91.75 amd64 Complete Generic Linux kernel and headers
ii linux-headers-4.4.0-91 4.4.0-91.114~14.04.1 all Header files related to Linux kernel version 4.4.0
ii linux-headers-4.4.0-91-generic 4.4.0-91.114~14.04.1 amd64 Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
ii linux-headers-generic-lts-xenial 4.4.0.91.75 amd64 Generic Linux kernel headers
ii linux-image-4.4.0-91-generic 4.4.0-91.114~14.04.1 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii linux-image-extra-4.4.0-91-generic 4.4.0-91.114~14.04.1 amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii linux-image-generic-lts-xenial 4.4.0.91.75 amd64 Generic Linux kernel image
ii linux-lts-xenial-cloud-tools-4.4.0-91 4.4.0-91.114~14.04.1 amd64 Linux kernel version specific cloud tools for version 4.4.0-91
ii linux-lts-xenial-tools-4.4.0-91 4.4.0-91.114~14.04.1 amd64 Linux kernel version specific tools for version 4.4.0-91
ii linux-tools-4.4.0-91-generic 4.4.0-91.114~14.04.1 amd64 Linux kernel version specific tools for version 4.4.0-91
ii linux-tools-virtual-lts-xenial 4.4.0.91.75 amd64 This package will always depend on the latest minimal generic kernel tools.

I hope this issue can get fixed soon. Stop making backups is not really an option for us.

[Dexuan Cui (decui)]

I read through the long bug log and found an interesting thing:

In #36, Andrey Vertexx (vertexx) reported the issue was fixed by the kernel in #30, but later Andrey thought the same kernel couldn't work any more?

In #47, #48, #54, #59 a lot of people , Aleksey (noirfry) , Khallaf (mkhallaf), Eric (jumpiem) etc., said removing the virtual DVD could resolve the issue or the boot issue? But it looks later this workaround can't work any more?

Please correct me if I don't get it right.

PS, I'm debugging a similar (or the same?) issue with RHEL 7.3 + LIS 4.2.2, and I happened to find this Ubuntu bug.

[James Bland (fastlanejb)]

For reference I've got this problem on Ubuntu 16.04 with the HWE kernel installed on VM's as well. I'd gone a long time using LXC and KVM, avoiding Hyper-V because of the VSS Read Only issue that took 2 years to fix. Noticed that was marked as fixed and almost straight away on first try of Hyper-V again hit this issue :(

Machine was on Dynamic memory but I've tried giving it 4GB of RAM fixed as well. Currently it's only got Docker and Portainer pretty much running on it and normally loads of free RAM.

Backups for machines start at 2am and the console I don't have an image of but is showing the same messages people are saying on the 4.4 kernel here.

Aug 16 02:03:14 kt-docker-01 kernel: [13629.889988] hv_utils: TimeSync IC version 4.0
Aug 16 02:03:14 kt-docker-01 kernel: [13629.890862] sd 0:0:0:0: [storvsc] Sense Key : Unit Attention [current]
Aug 16 02:03:14 kt-docker-01 kernel: [13629.890952] sd 0:0:0:0: [storvsc] Add. Sense: Changed operating definition
Aug 16 02:03:14 kt-docker-01 kernel: [13629.891023] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa
Aug 16 02:03:14 kt-docker-01 kernel: [13629.916480] sd 0:0:0:1: [storvsc] Sense Key : Unit Attention [current]
Aug 16 02:03:14 kt-docker-01 kernel: [13629.916544] sd 0:0:0:1: [storvsc] Add. Sense: Changed operating definition
Aug 16 02:03:14 kt-docker-01 kernel: [13629.916629] sd 0:0:0:1: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa
Aug 16 02:03:14 kt-docker-01 kernel: [13630.077401] scsi 0:0:1:0: Direct-Access Msft Virtual Disk 1.0 PQ: 0 ANSI: 5
Aug 16 02:03:14 kt-docker-01 kernel: [13630.162300] scsi host0: scsi_prep_async_scan called twice
Aug 16 02:03:14 kt-docker-01 kernel: [13630.250478] scsi host0: scsi_prep_async_scan called twice
Aug 16 02:03:14 kt-docker-01 kernel: [13630.310199] scsi host0: scsi_prep_async_scan called twice
Aug 16 02:03:14 kt-docker-01 kernel: [13630.372832] sd 0:0:1:0: [sdc] Sector size 0 reported, assuming 512.
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373303] sd 0:0:1:0: Attached scsi generic sg2 type 0
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373429] sd 0:0:1:0: [sdc] 1 512-byte logical blocks: (512 B/512 B)
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373434] sd 0:0:1:0: [sdc] 0-byte physical blocks
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373559] sd 0:0:1:0: [sdc] Write Protect is off
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373562] sd 0:0:1:0: [sdc] Mode Sense: 00 00 00 00
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373652] sd 0:0:1:0: [sdc] Asking for cache data failed
Aug 16 02:03:14 kt-docker-01 kernel: [13630.373733] sd 0:0:1:0: [sdc] Assuming drive cache: write through
Aug 16 02:03:14 kt-docker-01 kernel: [13630.374612] sd 0:0:1:0: [sdc] Sector size 0 reported, assuming 512.
Aug 16 02:03:14 kt-docker-01 kernel: [13630.375205] sd 0:0:1:0: [sdc] Attached SCSI disk
Aug 16 02:17:07 kt-docker-01 kernel: [14462.940527] sd 0:0:0:0: [storvsc] Sense Key : Unit Attention [current]
Aug 16 02:17:07 kt-docker-01 kernel: [14462.940579] sd 0:0:0:0: [storvsc] Add. Sense: Ch...

[James Bland (fastlanejb)]

Forgot to add to the last post but Kernel is:

Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.10.0-32-generic x86_64)

I've run the following commands on a clean install of Ubuntu 16.04 to upgrade it.

apt-get install -y --install-recommends linux-virtual-hwe-16.04
apt-get install -y linux-tools-virtual-hwe-16.04 linux-cloud-tools-virtual-hwe-16.04

[Dexuan Cui (decui)]

@fastlanejb are you on Windows Server 2012 R2 or 2016? Is your VM running some I/O intensive workload when the live backup happens? It looks you get the OOM issue every time you do the live backup? I'm digging into the issue, and trying to reproduce it first.

[Dave (furiousd)]

In our environment we're running windows 2016 datacenter with Ubuntu 16.04 VMs (latest kernel).

[Benjamin Knodel (b-knodel)]

Same Problem...

- Server 2012 R2 / Ubuntu 16.4.x with Kernel 4.4.0-62 (and virtual DVD-Drive) was fine for a long time with vm-backup via. Backup Exec.

- Server 2012 R2 / Ubuntu 16.4.x with updated Kernel to 4.4.0-81 (and virtual DVD-Drive) was crashing a few minutes after vm-backup with backupexec

- Server 2012 R2 / Ubuntu 16.4.x with updated Kernel to 4.4.0-83 (and virtual DVD-Drive) was crashing a few minutes after vm-backup with backupexec

- Server 2012 R2 / Ubuntu 16.4.x with updated Kernel to 4.4.0-83 (with removed virtual DVD-Drive) was running over some weeks without problems, but is now crashing after backup-exec-job.

[Werner Reinersmann (belgarethde)]

After "upgrading" (i. e. reinstall) to Ubuntu 17.04 server including the recommended packages from Microsoft <https://technet.microsoft.com/en-us/library/dn531029.aspx> my problems have disappeared.

[Markus Berg (onhosting)]

We have the same problems with Gen1 VMs on Hyper-V Cluster Windows 2016 Datacenter and Ubuntu 16.04.3 LTS 4.4.0-97.
Backup is done via DPM 2016.
As long hv-vss-daemon is active the VMs freeze during nighttime with a Out of Memory event.

[Dexuan Cui (decui)]

People are working on this issue: e.g. it looks the patch may work around it:
https://patchwork.kernel.org/patch/10012603/ (it would be great if somebody can test the patch)

Long will send one more patch:
https://<email address hidden>/msg1517902.html

[Andrey Vertexx (vertexx)]

The problem still here...
Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-97-generic x86_64) on Hyper-V 2012 R2

[Andrey Vertexx (vertexx)]

The problem still here...
Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-98-generic x86_64) on Hyper-V 2012 R2

[Jørgen Staun (staun)]

Randomly we experience the same on:

Ubuntu 16.04.03 LTS (GNU/Linux 4.4.0-98-generic x86_64)

Hyper-V 2016 /Veeam 9.5 update2

[Sitsofe Wheeler (sitsofe)]

I see this issue too. It is easy to get with a 4.4 kernel if you have an SSD hooked up to the Windows host and then you connect the SSD to the Hyper-V VM via the SCSI controller in passthrough mode. Error messages will look like this:

[ 20.476616] sd 3:0:0:0: [sdb] Synchronizing SCSI cache
[ 38.374458] scsi 3:0:0:0: Direct-Access Samsung SSD 850 PRO 256G EXM0 PQ: 0 ANSI: 0
[ 38.374703] scsi 3:0:0:1: Direct-Access PQ: 0 ANSI: 0
[ 38.374919] scsi 3:0:0:2: Direct-Access PQ: 0 ANSI: 0
[ 38.375368] scsi 3:0:0:3: Direct-Access PQ: 0 ANSI: 0
[ 38.375711] scsi 3:0:0:4: Direct-Access PQ: 0 ANSI: 0
[ 38.376526] scsi 3:0:0:5: Direct-Access PQ: 0 ANSI: 0
[ 38.376791] scsi 3:0:0:6: Direct-Access PQ: 0 ANSI: 0
[ 38.377114] scsi 3:0:0:7: Direct-Access PQ: 0 ANSI: 0
[ 38.392809] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 38.393282] sd 3:0:0:0: [sdb] 500118192 512-byte logical blocks: (256 GB/238 GiB)
[ 38.393811] sd 3:0:0:1: [sdc] Sector size 0 reported, assuming 512.
[ 38.393819] sd 3:0:0:1: [sdc] 1 512-byte logical blocks: (512 B/512 B)
[ 38.393822] sd 3:0:0:1: [sdc] 0-byte physical blocks
[ 38.393874] sd 3:0:0:1: Attached scsi generic sg3 type 0
[ 38.394262] sd 3:0:0:0: [sdb] Write Protect is off
[ 38.394267] sd 3:0:0:0: [sdb] Mode Sense: 0f 00 00 00
[ 38.394277] sd 3:0:0:1: [sdc] Write Protect is off
[ 38.394281] sd 3:0:0:1: [sdc] Mode Sense: 00 00 00 00
[ 38.395394] sd 3:0:0:2: [sdd] Sector size 0 reported, assuming 512.
[ 38.395402] sd 3:0:0:2: [sdd] 1 512-byte logical blocks: (512 B/512 B)
[ 38.395405] sd 3:0:0:2: [sdd] 0-byte physical blocks
[ 38.395553] sd 3:0:0:2: Attached scsi generic sg4 type 0
[ 38.395730] sd 3:0:0:2: [sdd] Write Protect is off
[ 38.395735] sd 3:0:0:2: [sdd] Mode Sense: 00 00 00 00
[ 38.395746] sd 3:0:0:1: [sdc] Asking for cache data failed
[ 38.395754] sd 3:0:0:1: [sdc] Assuming drive cache: write through
[ 38.400079] sd 3:0:0:2: [sdd] Asking for cache data failed
[ 38.400088] sd 3:0:0:2: [sdd] Assuming drive cache: write through
[ 38.400268] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 38.400418] sd 3:0:0:3: [sde] Sector size 0 reported, assuming 512.
[ 38.400425] sd 3:0:0:3: [sde] 1 512-byte logical blocks: (512 B/512 B)
[ 38.400427] sd 3:0:0:3: [sde] 0-byte physical blocks
[ 38.400480] sd 3:0:0:3: [sde] Write Protect is off
[ 38.400483] sd 3:0:0:3: [sde] Mode Sense: 00 00 00 00
[ 38.401653] sd 3:0:0:3: [sde] Asking for cache data failed
[ 38.401660] sd 3:0:0:3: [sde] Assuming drive cache: write through
[ 38.401768] sd 3:0:0:1: [sdc] Sector size 0 reported, assuming 512.
[ 38.402068] sd 3:0:0:2: [sdd] Sector size 0 reported, assuming 512.
[ 38.402258] sd 3:0:0:3: [sde] Sector size 0 reported, assuming 512.
[...]
[ 38.415473] sdi: partition table beyond EOD, enabling native capacity
[ 38.416509] sd 3:0:0:7: [sdi] Sector size 0 reported, assuming 512....

[Joshua R. Poulson (jrp)]

Interesting. Does this also reproduce with the linux-azure kernel, which is based on 4.11? (this should also work for on premises use, but you may have to install the "extra" package if you are using PCI passthrough).

[Sitsofe Wheeler (sitsofe)]

Perhaps in addition to the DVD fix 4.4 needs a few more of the patches listed on https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/log/drivers/scsi/storvsc_drv.c?h=v4.9.65 ?

[Sitsofe Wheeler (sitsofe)]

Joshua - I'm fairly certain it should be fixed in 4.11 - will reply when I have a chance.

[Sitsofe Wheeler (sitsofe)]

As predicted 4.11.0-1015-azure #15-Ubuntu doesn't have the problem and you see output like this:

[ 24.635325] sd 1:0:0:0: Attached scsi generic sg2 type 0
[ 24.638674] sd 1:0:0:0: [sdb] 500118192 512-byte logical blocks: (256 GB/238 GiB)
[ 24.641194] sd 1:0:0:0: [sdb] Write Protect is off
[ 24.641197] sd 1:0:0:0: [sdb] Mode Sense: 0f 00 00 00
[ 24.643644] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 24.653398] sd 1:0:0:0: [sdb] Attached SCSI disk
[ 24.741813] sd 1:0:0:0: [storvsc] Sense Key : Illegal Request [current]
[ 24.741818] sd 1:0:0:0: [storvsc] Add. Sense: Invalid command operation code
[ 24.743397] sd 1:0:0:0: [storvsc] Sense Key : Illegal Request [current]
[ 24.743400] sd 1:0:0:0: [storvsc] Add. Sense: Invalid command operation code

It is also broken in 4.8.0-46-generic #49~16.04.1-Ubuntu and fixed in 4.8.0-49-generic #52~16.04.1-Ubuntu and broken on 4.4.0-71-generic #92-Ubuntu and fixed in 4.4.0-75-generic #96-Ubuntu (driver history is http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/log/drivers/scsi/storvsc_drv.c?h=Ubuntu-4.4.0-75.96 in it) so it appears my previous 4.4.0 kernel simply wasn't new enough.

[Benjamin Knodel (b-knodel)]

Running some Hyper-V-VM´s with Ubuntu 16.04.3 with different Kernels and backed up via. Backup Exec 15

4.4.0-62-generic is fine
4.4.0-98-generic is fine
4.4.0-101-generic is fine

In the past I had problems with:
4.4.0-81-generic
4.4.0-83-generic

Installed LIS on every VM as follows :

apt-get install linux-virtual-lts-xenial
apt-get install linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial

https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-ubuntu-virtual-machines-on-hyper-v

BR

[Andrey Vertexx (vertexx)]

The problem still here...
Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-101-generic x86_64) on Hyper-V 2012 R2

[Eric (ezbg)]

same Problem here

Windows Server 2012 R2

16.04.3 LTS (Xenial Xerus)
4.4.0-104-generic

Only the VM's with more than 1 Disk freezes with out of memory. GEN1 and GEN2 doesn't matter in that case.

[Boris (bobster82)]

I can confirm: VMs with more than one disk still crash with out of memory. We're using Hyper-V on Windows Server 2012 R2 with GEN2 VMs (Ubuntu 16.04.3 LTS/Xenial, Kernel 4.4.0-104-generic). VMs with just one disk are running perfectly.

[gagzou (gagzou)]

Same: VMs with more than one disk still crash with out of memory. We're using Hyper-V on Windows Server 2012 R2 (core) with GEN2 VMs (Ubuntu 16.04 LTS/Xenial, Kernel 4.4.0-112-generic). VMs with just one disk are running perfectly.
Vss with 1disk ok
Vss with 2disk > freeze with out of memory...

[Никита (volonter)]

Same:
Use Windows Server 2012 R2, Hyper-V, GEN1 VM's.
VM: Ubuntu Server 16.04.03LTS, kernel - 4.4.0-112-generic, 2 disks.
linux-cloud-tools-virtual-lts-xenial 4.4.0.112.118
linux-tools-virtual-lts-xenial 4.4.0.112.118
VM down with out of memory.

[Arne Morten Johansen (arnemorten)]

I'm also having this exact same issue with the same specs as Никита.

2012 R2, Hyper-v, Gen1.
VM: Ubuntu Server 16.04.03LTS, kernel - 4.4.0-112-generic, 2 disks.
linux-cloud-tools-virtual-lts-xenial 4.4.0.112.118
linux-tools-virtual-lts-xenial 4.4.0.112.118

[Joshua R. Poulson (jrp)]

On 16.04, does this reproduce with the linux-azure kernel?

[Andrey Vertexx (vertexx)]

Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-124-generic x86_64)
The problem still here...

[Andrey Vertexx (vertexx)]

The problem still here...

[Boris (bobster82)]

Since 16.04.4 LTS with Kernel 4.4.0-128-generic/x86_64 the problems seems to be fixed. Our VM with more than one VHD are running without problems for two weeks now. Neither backups nor cluster failovers are crashing the VM.

[Jay Utah (bitsutah)]

I have had this exact problem with Ubuntu Server 16.04 thru 18.04 current on Citrix XenServer from version 7.1 thru 7.6 current.

Either HVM or PV, it doesn't seem to matter. When I live-migrate a Ubuntu Server VM, it seems like i have a 1/6 chance of the machine freezing within about a minute of the migration being done.

This migration roulette is awful when its the production database or NFS server. We've done a ton of XenServer updates thinking that would clear up the problem, but it seems that it isn't a XenServer problem.