Xenial server 16.04.x will have a black screen after PXE installation

Bug #1646277 reported by Mark W Wenning on 2016-11-30
12
This bug affects 3 people
Affects Status Importance Assigned to Milestone
debian-installer (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
linux (Ubuntu)
High
Joseph Salisbury
Xenial
High
Joseph Salisbury

Bug Description

The display will keep black screen after install 16.04.x via PXE(need use "Ctrl+Alt+F1" to change to terminal 1, then can see text mode screen)

Quanta team is installing Xenial server using PXE boot. After the installation, the console will show a black screen. Hitting ctl-alt-f1 displays the text mode screen OK.

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1646277/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
affects: ubuntu → debian-installer (Ubuntu)
Leo-Chen (leo-chen) on 2016-12-15
description: updated
Joseph Salisbury (jsalisbury) wrote :

Are there any call traces or errors in the logs after the installation? Once on the the other vt, does the system appear to function properly? If so, it might be good to test some other kernels to see if this is a regression, and test the latest mainline kernel to see if this is already fixed:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/

tags: added: kernel-da-key xenial
Changed in linux (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Changed in linux (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → Medium
Leo-Chen (leo-chen) wrote :

Hi
After update mainline kernel ,has fixed this problem
linux-headers-4.9.0-040900-generic_4.9.0-040900.201612111631_amd64.deb
linux-image-4.9.0-040900-generic_4.9.0-040900.201612111631_amd64.deb

Leo-Chen (leo-chen) wrote :

So..this issue, kernel team will release next Ubuntu 16.04.2 version with fix?

Joseph Salisbury (jsalisbury) wrote :

Thanks for testing. The bug will only be fixed if the commit in mainline that fixes the bug was also cc'd to stable.

We should probably perform a "Reverse" kernel bisect to identify the exact commit that fixes the bug. Then we will know if it in fact was cc'd to stable. If it was not, we can request it in stable and SRU it to Ubuntu.

We first need to identify the last bad kernel and first good one. Can you test the v4.9-rc1 kernel to try and narrow the affect versions down further. It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9-rc1/

If v4.9-rc1 is also good, we would want to test 4.8 final. If it is bad, we would want to test some of the newere 4.9 release candidates.

Thanks in advance!

Leo-Chen (leo-chen) wrote :

After update it OK,can boot into tty1 during boot into OS
linux-headers-4.9.0-040900rc1-generic_4.9.0-040900rc1.201610151630_amd64.deb
linux-image-4.9.0-040900rc1-generic_4.9.0-040900rc1.201610151630_amd64.deb

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in debian-installer (Ubuntu Xenial):
status: New → Confirmed
Changed in debian-installer (Ubuntu):
status: New → Confirmed
Leo-Chen (leo-chen) on 2016-12-28
summary: - Xenial desktop 16.04.x will have a black screen after PXE installation
+ Xenial server 16.04.x will have a black screen after PXE installation
Joseph Salisbury (jsalisbury) wrote :

Can you next test the 4.8 final kernel? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/

It might also be worthwhile to test the latest upstream 4.4 kernel. This will tell us if the fix was already sent to upstream stable. It is available from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.40/

Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Xenial):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
status: Triaged → In Progress
Changed in linux (Ubuntu Xenial):
status: Triaged → In Progress
tags: added: performing-bisect
Leo-Chen (leo-chen) wrote :

After update test the 4.8 was pass,it is work in tty1
Also update 4.4.40 same PASS

Joseph Salisbury (jsalisbury) wrote :

It is good news that 4.4.40 does not exhibit this bug. That might indicate the fix for the bug has already made it to upstream stable.

The current Ubuntu kernel in -proposed has up to the 4.4.35 updates. Can you give that kernel a test to see if the fix was applied to that version? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.35/

Leo-Chen (leo-chen) wrote :

That v4.4.35 was PASS

Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the -proposed kernel and post back if it resolves this bug?
See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed.

Thank you in advance!

Leo-Chen (leo-chen) wrote :

Already enabled proposed ,what are need to install kernel packages and update?

Joseph Salisbury (jsalisbury) wrote :

You should only have to run the following from a terminal:

sudo apt-get update
sudo apt-get dist-upgrade

Then reboot and run uname -a to ensure you are booted into the -proposed kernel.

Leo-Chen (leo-chen) wrote :

OK,The OS can boot into the proposed kernel after update and dist-updgrade

Joseph Salisbury (jsalisbury) wrote :

Thanks for testing. The kernel in -updates(4.4.0-59) now has the 4.4.35 updates. Can you apply the latest updates without using -proposed and confirm this bug is resolved?

Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Leo-Chen (leo-chen) wrote :

OS installer via PXE and then update to 4.4-59 kernel ,that still would stop at tty7 and I haven't looked into why it doesn't switch straight to a tty1 though
 update command as:
 sudo apt-get update
 sudo apt-get dist-upgrade

According to Leo, the issue is reproducible with 4.4.0-59. Changing status to confirmed.

Changed in linux (Ubuntu):
status: Fix Committed → Confirmed
Changed in linux (Ubuntu Xenial):
status: Fix Committed → Confirmed
Joseph Salisbury (jsalisbury) wrote :

The Xenial kernel is now at 4.4.0-64.85. Can you apply the latest updates and see if the bug still exists?

Leo-Chen (leo-chen) wrote :

After update to 4.4.0-64.85,that still would stop at tty7 and it doesn't switch straight to a tty1

Joseph Salisbury (jsalisbury) wrote :

Would you say this bug is now resolved?

NancyHsu (nancy-hsu) wrote :

After update to 4.4.0-64.85, the bug still exist.
It would stop at tty7 and it doesn't switch straight to a tty1.

Joseph Salisbury (jsalisbury) wrote :

It sounds like the upstream 4.4 kernel does not have the bug per comments #10 and #12.

It appears this bug may be caused by a SAUCE patch that is not in the upstream kernel. To confirm that, can you test the upstream 4.4.44 kernel, which is what 4.4.0-64.85 is based on?

If the bug does not exist in 4.4.44, we can perform a kernel bisect with the Ubuntu kernels to find that SAUCE commit that introduced this bug.

Joseph Salisbury (jsalisbury) wrote :

The 4.4.44 kernel can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.44/

NancyHsu (nancy-hsu) wrote :

Update to 4.4.44 kernel, the bug didn't happened.

It will import into which version of the official kernel or need to manually update patch kernal?

Joseph Salisbury (jsalisbury) wrote :

It sounded like this bug was not happening with -proposed in comment #16. Is that correct?

To perform a kernel bisect, we would need to know the last Ubuntu kernel version that was working and the first that is not. Do we know that information? If not, I can post some links to prior kernel versions for testing.

NancyHsu (nancy-hsu) wrote :

Yes, the bug was not happening with -proposed in comment #16.

On official kernel 4.4.0-31, we use below command to upgrade the newest kernel. The bug still happened.
sudo apt-get update
sudo apt-get dist-upgrade

Joseph Salisbury (jsalisbury) wrote :

The kernel in -updates is now 4.4.0-72.93. Are you only getting up to version 4.4.0-31 when you apply the updates?

NancyHsu (nancy-hsu) wrote :

On official kernel 4.4.0-31, we use below command upgrade to 4.4.0-72.93. The bug still exist.
sudo apt-get update
sudo apt-get dist-upgrade

Joseph Salisbury (jsalisbury) wrote :

Thanks for the response. Do you recall what kernel version you were running with the kernel in comment #16? The current kernel in -proposed is 4.4.0-73.94. However, comment #16 was back in January.

Leo-Chen (leo-chen) wrote :

Hi
We have tried more tested and verified,so far still not resolved this issue and but we have workaround them commented "# set vt_handoff=vt.handoff=7"in grub.cfg ,It would can switch straight tty1.however,Do you known what's mean about gfxmode ?
This issue always happened in PXE OS intaller
function gfxmode {
        set gfxpayload="${1}"
        if [ "${1}" = "keep" ]; then
               # set vt_handoff=vt.handoff=7
        else
                set vt_handoff=
        fi

NancyHsu (nancy-hsu) wrote :

Reply to comment #32.
We didn't remember what kernel version we were running with the kernel in comment #16.

Joseph Salisbury (jsalisbury) wrote :

It sounds like the best approach is to bisect through the Ubuntu Xenial kernels. We need to identify the last kernel version that did not have the bug and the first that did. Can you test the following early Xenial kernel:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414

Note, with this test kernel you need to install both the linux-image and linux-image-extra .deb packages.

NancyHsu (nancy-hsu) wrote :

We have already try many kernel version to check this bug.
And finally find 4.4.44 kernel didn't happened.
Please tell us which kernel version will be imported to fix the bug.
Thanks.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.44/

Joseph Salisbury (jsalisbury) wrote :

From all of the prior testing, this bug is happening only with Ubuntu kernels and not in any of the upstream kernels, such as 4.4.44.

We need to bisect through some older Ubuntu kernels to find the last good version and first bad version. This is the request posted in comment #35, to test this early version:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414

Joseph Salisbury (jsalisbury) wrote :

It might also be good to test the latest 17.04 kernel to see if the bug is already fixed there. It can be downloaded from:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/12405875

NancyHsu (nancy-hsu) wrote :

The link which you provide in comment #37, we can not find the link to download "amd64 build of linux 4.4.0-20.36 in ubuntu xenial RELEASE".

We try 17.04 kernel, the bug still not fixed.

Could you help to reproduce the bug with PXE server on your side? Thanks.

NancyHsu (nancy-hsu) wrote :

Is there anyone can help me to answer this problem?
Thanks.

tags: added: kernel-key
removed: kernel-da-key
tags: added: kernel-da-key
removed: kernel-key
tags: added: kernel-key
tags: removed: kernel-key

@nancy-hsu
Scroll down on https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414 you can find download link for
linux-image-4.4.0-20-generic_4.4.0-20.36_amd64.deb and linux-image-extra-4.4.0-20-generic_4.4.0-20.36_amd64.deb
Here are the links for your quick reference:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414/+files/linux-image-4.4.0-20-generic_4.4.0-20.36_amd64.deb
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414/+files/linux-image-extra-4.4.0-20-generic_4.4.0-20.36_amd64.deb

Thanks for your testing so far. All the upstream kernel (mainline) you have tested (4.9, 4.8, 4.4.40, 4.4.35 and 4.4.44) are fine, the two Ubuntu kernels 4.4.0-59 (based on upstream 4.4.35) and 4.4.0-64 (based on upstream 4.4.44) the bug can be reproduced. As #37, we need to find a working Ubuntu kernel to identify where the bug is first introduced, hence the request to test 4.4.0-20.

NancyHsu (nancy-hsu) wrote :

Download file:
linux-image-4.4.0-20-generic_4.4.0-20.36_amd64.deb
linux-image-extra-4.4.0-20-generic_4.4.0-20.36_amd64.deb

This issue can be reproduced on 4.4.0-20 kernel.
OS installer via PXE and then update to 4.4.0-20 kernel, that still would stop at tty7 and it doesn't switch straight to a tty1.

tags: added: kernel-key
removed: kernel-da-key
Changed in linux (Ubuntu Xenial):
importance: Medium → High
Changed in linux (Ubuntu):
importance: Medium → High
Joseph Salisbury (jsalisbury) wrote :

Can you see if this bug still exists with the 16.04.3 release? I'm thinking it should not, since prior testing showed that 4.9 and newer kernels did not have the bug.

16.04.3 is available from:
http://releases.ubuntu.com/16.04.3/

NancyHsu (nancy-hsu) wrote :

This bug still exists with 16.04.3.
Please help to check it, thanks.

Joseph Salisbury (jsalisbury) wrote :

Hi Nancy,

I'd like to confirm the symptoms of this bug. For PXE installs only, you get through all the installation steps and then the screen goes black after a reboot? You can then switch to tty1 and access command prompt?

NancyHsu (nancy-hsu) wrote :

Hi Joseph,

Yes, the symptoms are correct.
1. For PXE installs only.
2. Get through all the installation steps and then the screen goes black after a reboot.
3. Then can switch to tty1 and access command prompt.

Joseph Salisbury (jsalisbury) wrote :

And you are also able to reproduce this bug with the various test kernels I post? If so, we should be able to narrow this down without having to install every time.

NancyHsu (nancy-hsu) wrote :

Please take a look with #41.
All the upstream kernel (mainline) we have tested (4.9, 4.8, 4.4.40, 4.4.35 and 4.4.44) are fine.
The two Ubuntu kernels 4.4.0-59 (based on upstream 4.4.35) and 4.4.0-64 (based on upstream 4.4.44) the bug can be reproduced.
Finally we test kernel 4.4.0-20 the bug can also be reproduced.

Joseph Salisbury (jsalisbury) wrote :

Can you next see if this bug existed in 4.4.0-5:
https://launchpad.net/ubuntu/+source/linux/4.4.0-5.20/+build/8999514

NancyHsu (nancy-hsu) wrote :

Yes, the bug existed in 4.4.0-5.
Reproduce step:
1. PXE install Ubuntu 16.04.3.
2. Downgrade kernel to 4.4.0-5.
3. The screen goes black after a reboot.

Joseph Salisbury (jsalisbury) wrote :

Thanks for testing. Can you test this 3.19 kernel:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/13366909

This may not be a regression afterall, so we should figure that out.

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on kernel 3.19.0-93.
Test step:
1. PXE install Ubuntu 16.04.3.
2. Downgrade kernel to 3.19.0-93.
3. The screen goes normally after a reboot.

Joseph Salisbury (jsalisbury) wrote :

Thanks, can you next test 4.2.0-16:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/8099555

Also, this bug might be very related to graphics hw. Some server boards with remote management capabilities use something like Matrox or similarly exotic. In the past we sometimes saw problems switching framebuffer devices. Things that maybe can help are changing grub to console mode or adding nomodeset into kernel parameters. (If you change GRUB setting, be sure to run "sudo update-grub" before rebooting).

NancyHsu (nancy-hsu) wrote :

The bug existed in 4.2.0-16.
Reproduce step:
1. PXE install Ubuntu 16.04.3.
2. Downgrade kernel to 4.2.0-16.
3. The screen goes black after a reboot.

Joseph Salisbury (jsalisbury) wrote :

I built a 4.1 based test kernel. Can you see if this kernel exhibits the bug? It can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1646277/

NancyHsu (nancy-hsu) wrote :

The bug existed in 4.1.0-3 test kernel.
 Reproduce step:
 1. PXE install Ubuntu 16.04.3.
 2. Downgrade kernel to 4.1.0-3.
 3. The screen goes black after a reboot.

Joseph Salisbury (jsalisbury) wrote :

I built a 4.0 based test kernel. Can you see if this kernel exhibits the bug? It can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1646277/

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.0.0-4.
 Test step:
 1. PXE install Ubuntu 16.04.3.
 2. Downgrade kernel to 4.0.0-4.
 3. The screen goes normally after a reboot.

Joseph Salisbury (jsalisbury) wrote :

I started a kernel bisect between v3.7 final and v3.8-rc1. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
106c1ee52e54ee2a695dddfc86b4aed32f65894b

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.1.0-1.
 Test step:
 1. PXE install Ubuntu 16.04.3.
 2. Downgrade kernel to 4.1.0-1.
 3. The screen goes normally after a reboot.

Joseph Salisbury (jsalisbury) wrote :

I built the next test kernel, up to the following commit:
d9e5b220f962910fb37133bde7e857ce478daf3e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.1.0-1.
commit: d9e5b220f962910fb37133bde7e857ce478daf3e

Joseph Salisbury (jsalisbury) wrote :

I built the next test kernel, up to the following commit:
c6ff6112bd446505fb75a70690b953501bd4837f

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.1.0-3.
commit: c6ff6112bd446505fb75a70690b953501bd4837f

Joseph Salisbury (jsalisbury) wrote :

I built the next test kernel, up to the following commit:
5f54bebd73a3903514d7ab786085b21ea757f92b

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Hide

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.1.0-3.
commit: 5f54bebd73a3903514d7ab786085b21ea757f92b

Joseph Salisbury (jsalisbury) wrote :

I built the next test kernel, up to the following commit:
dddc618817c4eab6c1b3b1632fc47f2ed6c75fdc

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.1.0-3.
commit: dddc618817c4eab6c1b3b1632fc47f2ed6c75fdc

Joseph Salisbury (jsalisbury) wrote :

I built the next test kernel, up to the following commit:
ae827479a0a7a645ea96e68f5391fa71ad1e8c65

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

NancyHsu (nancy-hsu) wrote :

The bug didn't happened on test kernel 4.1.0-3.
commit: ae827479a0a7a645ea96e68f5391fa71ad1e8c65

Joseph Salisbury (jsalisbury) wrote :

I built the next test kernel, up to the following commit:
369e26773363a6c11186e2c5ec4d83a7c89fee9c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

tags: added: kernel-da-key
removed: kernel-key
NancyHsu (nancy-hsu) wrote :

The bug existed in 4.1.0-3 test kernel.
commint: 369e26773363a6c11186e2c5ec4d83a7c89fee9c
  Reproduce step:
  1. PXE install Ubuntu 16.04.3.
  2. Downgrade kernel to 4.1.0-3.
  3. The screen goes black after a reboot.

Joseph Salisbury (jsalisbury) wrote :

I built a test kernel with a revert of commit 369e26773363a6c11186e2c5ec4d83a7c89fee9c.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not?

NancyHsu (nancy-hsu) wrote :

The bug existed in 4.1.0-3 test kernel.
 commint: 369e26773363a6c11186e2c5ec4d83a7c89fee9c.(#73)
   Reproduce step:
   1. PXE install Ubuntu 16.04.3.
   2. Downgrade kernel to 4.1.0-3.
   3. The screen goes black after a reboot.

Joseph Salisbury (jsalisbury) wrote :

With a current kernel, can you try changing grub to console mode or adding nomodeset into kernel parameters.

After you change a GRUB setting, be sure to run "sudo update-grub" before rebooting.

NancyHsu (nancy-hsu) wrote :

Hi Joseph,

With a current kernel, that adding nomodeset into kernel parameters.
After change a GRUB setting then run "sudo update-grub" before rebooting.

The bug still existed.
The screen goes black after a reboot.

Joseph Salisbury (jsalisbury) wrote :

Did the bug go away with the linux-image-4.1.0-3-generic_4.1.0-3.3~lp1646277Commit369e26773Reverted_amd64.deb kernel posted in comment #73?

NancyHsu (nancy-hsu) wrote :

The bug still existed with the linux-image-4.1.0-3-generic_4.1.0-3.3~lp1646277Commit369e26773Reverted_amd64.deb kernel posted in comment #73.

Joseph Salisbury (jsalisbury) wrote :

The bisect must have gone wrong. It might not be possible to perform a bisect in this case. I think what would be best is to post all the HW info into the bug. Then I can review the commits between Ubuntu-4.0.0-4.7 and Ubuntu-4.1.0-3.3 to see what could be relevant.

Can you run the following command and then attach the report.1646277 file:
apport-bug --save /tmp/report.1646277 linux

Varun Priolkar (varunpriolkar) wrote :

I am still hitting this issue on the latest HWE as well as the normal kernel on 16.04. This is a major blocker for automating installs.

Varun Priolkar (varunpriolkar) wrote :

Got it to work by following instructions mentioned on this bug report:

https://bugs.launchpad.net/ubuntu/+source/grub-installer/+bug/1616258

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers