[Hyper-V] VM with ubuntu 32bit with linux-next does not boot

[Chris Valean]

We identified that at least ubuntu (let's take 15.10) does not boot if installed in the 32bit architecture, and using the linux-next upstream kernel 4.5 on it. This has been seen on other distributions as well.

We took both the official linux-next branch from kernel.org and the one from kernel.ubuntu.com v4.5-rc7.
For the compilation process we went either with the 4.2 kernel config file from the installed running kernel, and also built the config separately.
No errors have been encountered during the build or install process of the 4.5 kernel, however at boot, the VM will just hang.
There are no call traces or messages at boot to show any pottential issue.

Have you seen this on your end when testing the 32bit build?
Attaching the full serial log for reference, however I don't see any errors in the kernel boot process.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1555786

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

I can confirm that 4.5 final and 4.5-rc7 fail to boot on a 32bit guest on my setup as well. I am seeing an I/O error on the disk. Booting back into a 15.10 kernel, the disk is automatically fsck'd and boots fine. Going back to a mainline kernel, I get the I/O error again.

I'll test some of the other 4.5 release candidates to find the last good one and first bad one. We can then bisect to find the commit that introduced this.

[Chris Valean (cvalean)]

Thanks Joe for the confirmation!
This issue was first seen as back as 4.4 rc8 when we picked it up first, so we have to go even back in time with the releases to find the problem.
Question comes then if this pre-dates 4.4 stable, would this help with the bisect, since 4.4 is used in 16.04.

We've also took the latest Xenial i386 and compiled linux-next (4.5-20160318) with the .config from the 4.4 kernel that comes with 16.04.
Although the kernel compile went without errors, the new kernel boot process is still unsuccessful.

[Joseph Salisbury (jsalisbury)]

I tested all the upstream kernels going back to 4.2. None of them will boot in a 32 bit guest. I had not problem creating a new Xenial 32 bit guest, which is based on the 4.4 kernel. However, the system would not boot the upstream 4.4 kernel. It may be that Ubuntu has a SAUCE patch applied that is not in upstream yet.

Are you able to boot any of the other upstream kernels, such as 4.2, 4.3, 4.4 ?

[DorinBarboiu (dbarboiu)]

Tested with upstream kernel 4.0 and 4.0.9 on a 32 bit VM. With both of this version VM booted as expected without issues.
Also I tested with upstream kernel 4.1 and VM did not boot. I expect that the issue is present for first time in upstream kernel 4.1.

My question is: Could you please, help me to bisect this issue?

[Joseph Salisbury (jsalisbury)]

Sure, I can assist with a bisect.

I can either build kernels and post them for you to test, or I can test them myself on my Hyper-V setup. My setup is currently tied up running test until probably Monday for bug 1470250. Just let me know if you want me to start building test kernels for you. Otherwise, I'll start testing for this bug next week.

[Chris Valean (cvalean)]

Hi Joe,
You can send us the test kernels and we can quickly verify them.

Thank you!

[Joseph Salisbury (jsalisbury)]

Hi Dorin,

Can you test the v4.1-rc1 kernel to see if that was the release candidate that introduced the regression? If it is not, we should test some of the other release candidates until we find the last good one and first bad one.

The 4.1-rc1 kernel can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc1-vivid/

[DorinBarboiu (dbarboiu)]

Hi Joseph,

Tested with this kernel http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0.9-wily/ to be sure that the VM is booting as expected.
Also when I tested with this kernel http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc1-vivid/ the VM did not boot.

Thanks.

[Chris Valean (cvalean)]

Hi Joe,

Have you had the chance to do some bisect on the patch from 4.0.9 to 4.1?

[Joseph Salisbury (jsalisbury)]

Sorry, the first test kernel for the bisect failed. I'll dig into the failure and get the first test kernel shortly.

[Joseph Salisbury (jsalisbury)]

Sorry for the delay. I figured out the build failure for the bisect.

I built the first test kernel, up to the following commit:
6c373ca89399c5a3f7ef210ad8f63dc3437da345

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1556875

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Chris Valean (cvalean)]

Hi Joe,

The link provided is not valid. I also checked all the folders from your repo dated 13-May-2016 but none seem to have the 32bit packages.

[Joseph Salisbury (jsalisbury)]

Sorry, I posted the 64 bit .debs. The 32 bit packages should now be posted here:

http://kernel.ubuntu.com/~jsalisbury/lp1555786/

[DorinBarboiu (dbarboiu)]

Hi Joe,

Tested on Ubuntu Xenial 32 bit and Ubuntu 14.04.3 32 bit and the bug is not present. VM booted very well.

Thanks.

[Joseph Salisbury (jsalisbury)]

I built the first test kernel, up to the following commit:
2c33ce009ca2389dbf0535d0672214d09738e35e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1556875

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Chris Valean (cvalean)]

lp1556875 is not a valid folder, would it be lp1555786 again where you put the latest deb files?

[Joseph Salisbury (jsalisbury)]

Yes, sorry. Cut and paste error. The correct folder is:

http://kernel.ubuntu.com/~jsalisbury/lp1555786/

[Chris Valean (cvalean)]

I tried the 4.0.0-040000.201605171131_i386 kernel however it will hang at boot, so we might be getting closer to the root patch.

[Joseph Salisbury (jsalisbury)]

I built the first test kernel, up to the following commit:
4fc8adcfec3da639da76e8314c9ccefe5bf9a045

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1556875

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Chris Valean (cvalean)]

The 4.0.0-040000-generic_4.0.0-040000.201605191143_i386 kernel from lp1555786 also does *not* boot.

[Chris Valean (cvalean)]

We have a patch that was created for 32bit and testing it now it's also resolving the upstream 32bit boot issue.
Waiting for it to be accepted in upstream, so we should be done with any of the current bisect work.

[Joseph Salisbury (jsalisbury)]

Do you want me to build a test kernel with your new patch now, or do you want to wait until it lands upstream?

[Joshua R. Poulson (jrp)]

Wait for our upstream submission, please. There may be feedback.

[Joshua R. Poulson (jrp)]

https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=68ab2d76e4be785a7003fdb42b7c4ed8bba56ae2

[Joseph Salisbury (jsalisbury)]

Testing in comment #5 confirmed this bug was only happening with the upstream kernels and not Ubuntu kernels. Is that still the case with Xenial?

Also, can you confirm the patch in comment #26 is needed in Yakkety and Zesty to get 32 bit Ubuntu VMs to boot? If so, I will submit it for SRU to whatever releases need it.

[Dexuan Cui (decui)]

@jrp, it looks the patch in comment #26 is unrelated to this bug?
The patch is for cxlflash (Support for IBM CAPI Flash), which doesn't exist in a VM running on Hyper-V.

[Chris Valean (cvalean)]

Closing item, this got resolved in upstream eventually.

[Andy Whitcroft (apw)]

This bug was nominated against a series that is no longer supported, ie yakkety. The bug task representing the yakkety nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.