[Microsoft Hyper-V guest] System shows graphic artifacts for a moment, then text cursor for about minute and then starts

Bug #1848534 reported by M
36
This bug affects 6 people
Affects Status Importance Assigned to Milestone
gdm3 (Ubuntu)
Confirmed
Undecided
Unassigned
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

AFter upgrade system shows graphic artefacts for a moment and then text cursor for about minute (it looks like hanged) and then starts.

In 19.04 startup required 1 or 2 seconds.

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: ubuntu-release-upgrader-core 1:19.10.15
ProcVersionSignature: Ubuntu 5.3.0-18.19-generic 5.3.1
Uname: Linux 5.3.0-18-generic x86_64
ApportVersion: 2.20.11-0ubuntu8
Architecture: amd64
CrashDB: ubuntu
CurrentDesktop: ubuntu:GNOME
Date: Thu Oct 17 17:42:27 2019
InstallationDate: Installed on 2019-07-04 (104 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
PackageArchitecture: all
SourcePackage: ubuntu-release-upgrader
Symptom: dist-upgrade
UpgradeStatus: Upgraded to eoan on 2019-10-17 (0 days ago)
VarLogDistupgradeLspcitxt:

VarLogDistupgradeXorgFixuplog:
 INFO:root:/usr/bin/do-release-upgrade running
 INFO:root:No xorg.conf, exiting
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu8
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
BootLog: Error: [Errno 13] Permission denied: '/var/log/boot.log'
CompositorRunning: None
CurrentDesktop: ubuntu:GNOME
DistUpgraded: 2019-10-17 17:03:47,139 DEBUG Running PostInstallScript: './xorg_fix_proprietary.py'
DistroCodename: eoan
DistroRelease: Ubuntu 19.10
DistroVariant: ubuntu
ExtraDebuggingInterest: Yes
GraphicsCard:

InstallationDate: Installed on 2019-07-04 (105 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
Package: xorg-server (not installed)
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.3.0-18-generic root=UUID=17409d40-25e9-4051-9fd9-758e2a02ebc3 ro quiet splash video=hyperv_fb:1900x900 elevator=noop vt.handoff=7
ProcVersionSignature: Ubuntu 5.3.0-18.19-generic 5.3.1
RelatedPackageVersions:
 linux-restricted-modules-5.3.0-18-generic N/A
 linux-backports-modules-5.3.0-18-generic N/A
 linux-firmware 1.183
RfKill:

Tags: eoan ubuntu
Uname: Linux 5.3.0-18-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-10-17 (0 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 01/30/2019
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v4.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v4.0
dmi.chassis.asset.tag: 2831-3616-6111-5725-4803-1162-28
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v4.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.0:bd01/30/2019:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.0:
dmi.product.family: Virtual Machine
dmi.product.name: Virtual Machine
dmi.product.sku: None
dmi.product.version: Hyper-V UEFI Release v4.0
dmi.sys.vendor: Microsoft Corporation
version.compiz: compiz N/A
version.libdrm2: libdrm2 2.4.99-1ubuntu1
version.libgl1-mesa-dri: libgl1-mesa-dri 19.2.1-1ubuntu1
version.libgl1-mesa-glx: libgl1-mesa-glx N/A
version.xserver-xorg-core: xserver-xorg-core 2:1.20.5+git20191008-0ubuntu1
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev N/A
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:19.0.1-1ubuntu1
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.917+git20190815-1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.16-1

Revision history for this message
M (xh-parcen-et) wrote :
affects: ubuntu-release-upgrader (Ubuntu) → xorg (Ubuntu)
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Please run this command:

  lspci -k > lspcik.txt

and then send us the file 'lspcik.txt'.

Please also attach the log files /var/log/Xorg*.log

summary: - System shows graphic artifacts for a moment, then text cursor for about
- minute and then starts
+ [Microsoft Hyper-V guest] System shows graphic artifacts for a moment,
+ then text cursor for about minute and then starts
affects: xorg (Ubuntu) → xorg-server (Ubuntu)
Changed in xorg-server (Ubuntu):
status: New → Incomplete
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

The kernel log shows a long delay but it's unclear what that delay is:

[ 3.907379] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 49.467099] hv_balloon: Max. dynamic memory size: 17000 MB
[ 107.602021] Lockdown: Xorg: ioperm is restricted; see man kernel_lockdown.7
[ 111.600955] rfkill: input handler disabled

Please reboot, reproduce the problem again and then run:

  journalctl -b0 > journal.txt

and attach the file 'journal.txt' here.

Revision history for this message
M (xh-parcen-et) wrote :

One thing, which I didn't mentioned earlier:

there is graphic artifact (color on half of screen), then "Ubuntu loading" dots, then text cursor with delay, then login screen.

I forgotten about "Ubuntu loading" dots.

lspci -k output is empty (even with sudo), the rest is attached.

Revision history for this message
M (xh-parcen-et) wrote :

And one more log

Changed in xorg-server (Ubuntu):
status: Incomplete → New
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1848534

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
M (xh-parcen-et) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected ubuntu
description: updated
Revision history for this message
M (xh-parcen-et) wrote : CRDA.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : CurrentDmesg.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : DpkgLog.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : MonitorsUser.xml.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : ProcEnviron.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : ProcInterrupts.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : ProcModules.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : PulseList.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : UdevDb.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : WifiSyslog.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : XorgLog.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : XorgLogOld.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : Xrandr.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote : xdpyinfo.txt

apport information

Revision history for this message
M (xh-parcen-et) wrote :

I've tried to (1) boot the same machine with kernel 5.0.0.32 (2) create new machine without upgrading from 19.04

Results were the same, anybody?

Revision history for this message
Dexuan Cui (decui) wrote :

Hi M, I can not reproduce the issue: just now I downloaded http://dt0cinyuc0rrg.cloudfront.net/ubuntu-19.10-desktop-amd64.iso and created a Generation-2 VM on Hyper-V with the .iso file. The VM boots up fast: it boots up to the Xorg desktop in 28 seconds with 1 CPU and 2GB memory, and in 21 seconds with 8 CPUs and 8GB memory.

I don't see anything unusual.

I believe you're also using Generation-2 VM since your "lspci" returns nothing.

FYI: I got the below in my VM:

decui@decui-u1910-gen2:~$ uname -a
Linux decui-u1910-gen2 5.0.0-32-generic #34-Ubuntu SMP Wed Oct 2 02:06:48 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

decui@decui-u1910-gen2:~$ dmesg | grep "Hyper-V Host Build"
[ 0.000000] Hyper-V Host Build:18928-10.0-1-0.1044

decui@decui-u1910-gen2:~$ systemd-analyze
Startup finished in 303ms (firmware) + 663ms (loader) + 1.087s (kernel) + 35.041s (userspace) = 37.096s
graphical.target reached after 34.963s in userspace

decui@decui-u1910-gen2:~$ systemd-analyze blame
         31.530s plymouth-quit-wait.service
         10.598s gdm.service
          1.817s dev-sda2.device
           944ms networkd-dispatcher.service
           866ms NetworkManager-wait-online.service
          .....

decui@decui-u1910-gen2:~$ systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @34.963s
└─multi-user.target @34.963s
  └─kerneloops.service @4.213s +34ms
    └─network-online.target @4.170s
      └─NetworkManager-wait-online.service @3.301s +866ms
        └─NetworkManager.service @2.983s +317ms
          └─dbus.service @2.912s
            └─basic.target @2.765s
              └─sockets.target @2.765s
                └─snapd.socket @2.754s +10ms
                  └─sysinit.target @2.731s
                    └─apparmor.service @2.199s +530ms
                      └─local-fs.target @2.173s
                        └─run-user-1000-gvfs.mount @17.395s
                          └─run-user-1000.mount @14.119s
                            └─local-fs-pre.target @456ms
                              └─keyboard-setup.service @223ms +232ms
                                └─systemd-journald.socket @218ms
                                  └─-.mount @206ms
                                    └─system.slice @206ms
                                      └─-.slice @206ms

Revision history for this message
Dexuan Cui (decui) wrote :

I'm not sure what exact issues you're reporting.

Your VM takes too much time to boot up? How long? "systemd-analyze blame" should collect the info for your VM.

Your VM's screen is somehow messed up temporarily during the boot-up process? Or the boot-up process is stuck in the "text cursor" screen for a long period of time (about 1 minute?) and you'd like to figure out what's happening during that period of time? But since it looks your VM is able to boot up in 2 minutes or so (?), it looks there is no fatal issue?

You're saying you can reproduce the issue with a fresh VM created from the .iso file (ubuntu-19.10-desktop-amd64.iso)?

It would be helpful if you can share your output of the same commands I ran.

A video of your VM's boot-up process would be helpful, if that's not difficult. I don't know how you would share the video. :-)

Revision history for this message
M (xh-parcen-et) wrote :

Hi,

Thanks for comments #25 and #26,

I don't know how fast is your machine, but on mine earlier (before 19.10) I had startup in just few seconds, now have this delay. Yes, this is maybe not critical, but should be fixed, because can go into Ubuntu 20.04 ;)

Anyway, I'm using stock kernel 5.3 + I've done "systemd-analyze blame"

In first line I had also many seconds with plymouth-quit-wait.service.

I've done

sudo apt remove plymouth-theme-ubuntu-text
sudo apt remove plymouth-theme-ubuntu-logo
sudo apt remove lighttpd

and removed splash from /etc/default/grub

(yes, I can do it, this is not my critical VM)

Result:

kernel starts, I see messages, then screen cleans and I have delay

systemd-analyze doesn't show anything specific (all delays are very short - the longest is gdm like yours and the rest looks quite similar)

I don't know if this is Ubuntu or Hyper-V specific,

other ideas?

Revision history for this message
Dexuan Cui (decui) wrote :

Hi M, since I can not reproduce the delay issue, I don't know what I can do now. :-(

Do you think if it's related to Xorg?

Can you install a new VM from scratch from the server .iso (http://releases.ubuntu.com/19.10/ubuntu-19.10-live-server-amd64.iso) and see if you can reproduce the same issue?

The server .iso doesn't install Xorg, and a lot of other packages used in a Desktop environment. I hope you can not repro the issue with it, then we'll have a good starting point.

Revision history for this message
M (xh-parcen-et) wrote :

Hi,

If possible, please try 19.04 or LTS 18.04 and compare startup time - my guess is that it will start almost immediately and you will see, that you already reproduced it :)

I'm sure let's say in 90% :) -> 28 seconds is looonnnggg (like I said, I've got about 2 seconds for older Ubuntu).

Revision history for this message
Dexuan Cui (decui) wrote :

The typical boot-up time of my Ubuntu VM on Hyper-V is 20~30 seconds for a Desktop version of Ubuntu, and 10~20 seconds for a Server version. I tried Ubuntu 19.04 just now and it also took 20+ seconds.

I never achieve a boot-up time of 2s.

I do know Ubuntu can boot up fast in 2~3 seconds in WSL (Windows Subsystem for Linux), though I didn't try it in person.

Revision history for this message
M (xh-parcen-et) wrote :

Dear Ubuntu devs,

issue was reported on 17.10 and still - Ubuntu 19.04 starts fast, Ubuntu 19.10 upgraded from 19.04 / installed from scratch starts very slow.

ANY IDEAS?

Revision history for this message
Dexuan Cui (decui) wrote :

Can Ubuntu devs please try to repro the issue? I can not repro it. :-(

Hi M, I assume you can also repro the issue with a VM created from scratch from the server .iso (see comment #28) with a minimal installation? If yes, can you please share the vhdx file? If you configure the disk size to 15GB an use xfs (rather than ext4) in the installation process, the generated vhdx file should be 1.5GB or so (IIRC), so I guess there might be a way for you to share the file somewhere for me to download? Please also use less CPUs (e.g. 2) and memory for the VM (e.g. 2GB), if this doesn't prevent you from reproducing the issue.

Revision history for this message
M (xh-parcen-et) wrote :

Hi Dexuan,

Sorry for delay in answer - I have installed fresh Server 19.10 (installation is of course different - EFI partition setup by installer, etc. etc.) and after kernel boot messages I've got immediately login prompt.

After that I've done "sudo apt-get install ubuntu-desktop" and issue is visible.

size of VM disk with xfs 2,5 GB, with xfs + ubuntu-desktop 5,5GB , size of "normal" installation 8,5 GB

What can you propose in this situation?

Revision history for this message
Dexuan Cui (decui) wrote :

So let me summarize your findings on the same host of yours (I suppose your VMs use the same config for the number of vCPUs and the memory size. I also suppose you only tested Hyper-V Generation 2 VMs or you confirmed Gen-1 vs. Gen-2 makes no difference):

("fast" means you can see the GUI desktop or the text terminal prompt in about 1~2 seconds, and "slow" means you need a much longer time, e.g. 1 minute (?))

fresh Server 19.10 ==> fast
fresh Server 19.10 + the ubuntu-desktop package ==> slow
fresh Desktop 19.10 ==> slow
fresh Desktop 19.04 ==> fast
fresh Desktop 19.04 upgraded to 19.10 ==> slow

So it looks a change in 19.10 with the xorg causes the slowness.

However, I can not reproduce the issue, because both my fresh 19.10 and 19.04 VMs boot up in 20+ seconds and I never have a boot-up time of 1~2 seconds.

Hi M, can you please check this case:

fresh Desktop 19.04 upgraded to 19.10 ==> slow

What if you boot the VM with the 19.04 kernel + 19.10's userspace (including Xorg)?
If it's also slow, then we have more confidence that the 19.04 Xorg has an issue.
If it's fast, then the issue may be more likely that the interaction between the 19.10 Xorg and the 19.10 kernel is causing the issue.

Can you also please try booting the VM with a "good" 19.04 VM but (ONLY) upgrading the kernel to 19.10?

In the slow cases, can you check the logs files (/var/log/Xorg*, /var/log/syslog*) and see if there is any obvious error/warning?

Revision history for this message
Timo Aaltonen (tjaalton) wrote :

it's not X to blame, but the one that starts it so moving to gdm3 hoping that someone would have a clue

affects: xorg-server (Ubuntu) → gdm3 (Ubuntu)
Revision history for this message
M (xh-parcen-et) wrote :

Anybody? Or will be it just ignored ?

(info from #34 looks OK)

M (xh-parcen-et)
Changed in linux (Ubuntu):
status: Incomplete → New
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1848534

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
M (xh-parcen-et) wrote :

apport info was sent on 18 Oct...

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gdm3 (Ubuntu):
status: New → Confirmed
Revision history for this message
Matthew Swift (msgallery) wrote :

I reproduced this with Win10 x64 up to date on 4/11/2020 and the stock "quick create" versions of Ubuntu 18.04 and 19.10 as referenced and selected from the Hyper-V manager. Default options all the way. No updates installed after initial installation. This machine has AMD Ryzen 7 3800X and 32GB memory, NVMe SSD system drive -- it's fast. Default network adapter for the VMs. ipv6 enabled or disabled on underlying NIC and the vEthernet default switch (I tried both ways). 18.04 goes from "start" to X login in 11 seconds, including having to press through a console "error: no such device: root / press any key to continue" glitch. 19.10 goes from start to a Grub menu in about 3 seconds. 11 more seconds through console message about Spectre mitigation, then the Ubuntu splash screen of 5 dots on purple background, then console cursor holds for about 1:40. Then X login finally. System load during the long delay on network, disk, CPU under 5% during this time. Then suddenly a flurry of console messages followed by X login window. Seems very clear that the boot process is waiting for a long failed timeout somehow before proceeding.

By accident I discovered the following with 19.10. Select "system startup" from the grub menu. Result is Windows logo screen "Hyper-V UEFI / No boot devices were found." The "restart" button is not functional, have to "power off" the VM because "shutdown" fails. Re-power on without closing the connection window, and 19.10 proceeds through same sequence from "start" to X login in about 12 seconds: spectre mitigation message on console, Ubuntu splash, console cursor, then possibly the flurry of console messages (but so fast invisible), then X login. I can get through the sequence above of grub "system startup" then power-off, then restart to the X login from scratch in 25 seconds, whereas about 120 seconds if I just start and wait through the timeout.

Revision history for this message
Dexuan Cui (decui) wrote :

Today I installed a Generation-2 VM (4 virtual CPUS, 4 GB memory) from the this .iso file:
http://releases.ubuntu.com/19.10/ubuntu-19.10-desktop-amd64.iso.

My host is Win10: Version 1909 (OS Build 18383.720) -- I got the info by running the built-in "winver.exe" program.

The CPU type is Intel Core I7-7600 (2.80G Hz). There are 2 cores and the cores support SMT, so there are 4 logical processors in total.

I can see the "graphic artifact" (I will upload a screenshot soon), but it looks overall the boot-up is fast (it takes 30 seconds) and it looks the VM works fine for me.

When the VM boots up:
1. First, the screen with the purple background (I think it's from grub) remains 4 seconds.
2. The screen background becomes black, and there is a "Hyper-V" logo in the center of the screen. This remains about 1 second.
3. The screen with the "graphic artifact" appears, and remains about 4 seconds.
4. The screen background becomes purple and the "Ubuntu" logo with 5 dots appears. This screen remains 8 seconds.
5. The screen becomes completely black. This screen remains 9 seconds.
6. The screen becomes kind of purple again, and in about 2 seconds the GUI desktop appears (I set Ubuntu to automatically login in to the desktop).

So the overall time spent on the 6 steps are 30 seconds. IMO this looks normal.

Revision history for this message
Dexuan Cui (decui) wrote :

This is the screenshot of the graphic artifact mentioned in the previous comment.

Revision history for this message
Dexuan Cui (decui) wrote :

BTW, my Linux kernel version is 5.3.0-46-generic #38-Ubuntu (17:37:05, 3/27/2020).

The "graphic artifact" is somehow caused by the "$vt_hanoff" kernel parameter (check "cat /proc/cmdline").
If I manually remove the "$vt_hanoff" at the grub screen, I won't see the "graphic artifact" -- Ubuntu guys should take a look and fix the issue, as I'm not familiar with "vt_handoff".

@msgallery: I never see the "1:40" (1 minutes and 40 seconds) delay reported in comment #40. Maybe you can use "systemd-analyze critical-chain" (mentioned in Comment #25) to figure out why the delay happened.

To recap, my experience with the fresh Desktop installation of Ubuntu 19.10 (Gen-2 VM) on Hyper-V is good, except for the minor "graphic artifact" issue. I don't see any long delay.

Revision history for this message
Dexuan Cui (decui) wrote :

@msgallery: BTW, you mentioned 'The "restart" button is not functional' -- actually it is not functional only when we try to click the button by mouse -- if we press Tab to focus on the button and then press Enter, the VM should reboot. :-) I'll try to mention this to Hyper-V team, but I'm not sure when they will fix this minor issue, since the issue should be of low priority.

Revision history for this message
Matthew Swift (msgallery) wrote :
Download full text (7.8 KiB)

I've created a 2m35s desktop video showing a boot of stock 18.04 and 19.10 on my system, posted at https://chaetura.net/ms-vid1-bug-1848534.webm (18MB, renders in Chrome window for me, or use VLC to watch).

I've posted a second video showing the shortcut to boot 19.10 quickly that I described earlier, but which I have simplified to simply pulling the plug with "power off" from Hyper-V at the grub menu, do not close the Connection window, restart, and 19.10 will boot fast. Note that "system setup" (I previously mistakenly wrote "system startup") doesn't appear in the grub menu of stock 19.10; it appears only after updating packages in Ubuntu.

Furthermore, the shortcut boot of 19.10 solves *THREE* significant problems with stock 19.10 that are not problems with stock 18.04.3:

    1) avoids the long delay in startup
    2) allows user to select any screen size for the Connection (whereas stock 19.10 came in one size only
       and user has no opportunity to select; goes hand in hand with the different login screen, as you

I see that graphical artifact at some point during boot of any Ubuntu VM in Hyper-V. You can see it in the video too. In 19.10 it comes while the Spectre mitigation message is on the screen in console, at a point where console changes its rendering/mode somehow and the message reappears in a slightly different font.
I forgot to show /proc/cmdline and "uname -a" in the video but for 19.10 it is:

   BOOT_IMAGE=/boot/vmlinuz-5.3.0-23-generic root=PARTUUID=[blahblah] ro quiet splash
   Linux stock19 5.3.0-23-generic #25-Ubuntu SMP Tue Nov 12 09:22:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

and in 18.04.3 it is

    BOOT_IMAGE=/boot/vmlinuz-5.0.0-25-generic root=LABEL=desktop-rootfs ro quiet splash vt.handoff=1
    Linux stock18 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Note that I installed my 19.10 via Hyper-V Manager's "quick-create" rather than from an .iso downloaded separately. Same kernel version, but mine is an earlier packaging number, and who knows what else may be different about the install image. The manager says the image was updated 17 October 2019. The Manager downloads a zip file from partner-images.canonical.com via https so I cannot snoop the URL looking at the packets. I have a copy of the .vhdx which is insie the zip file, and I'm messing around with trying to mount it in another VM but I haven't solved that yet.

My Windows 10 Pro x64 build (today) is Win10: Version 1909 (OS Build 18363.778). No further windows updates are available, and I don't understand Windows version numbers, so I can't explain the discrepancy from yours.

The startup delay persists after updating the stock install of 19.10 (using aptitude; all available updates accepted but the grub packages held back till I learn how to do them; they broke my last install).

I noticed while making the video that during the long startup delay, the process is using GPU at 5% -- significant amount of usage (documented in video). Therefore I've included my GPU information at bottom below.

I boot stock quick-create 19.10. I change the name of the VM only ("Stock19"). Default ethernet adapter. ...

Read more...

Revision history for this message
Matthew Swift (msgallery) wrote :

A stray click sent my previous message before I had finished editing it, and I see no way to edit my post. I will post fully complete/edited version momentarily. I hope an admin will delete this message and my prior message to avoid cruft in this thread.

Revision history for this message
Matthew Swift (msgallery) wrote :
Download full text (7.1 KiB)

I've created a 2m35s desktop video showing a boot of stock 18.04 and 19.10 on my system, posted at https://chaetura.net/ms-vid1-bug-1848534.webm (18MB, renders in Chrome window for me, or use VLC to watch).

I've posted a second video showing the shortcut to boot 19.10 quickly that I described earlier, but which I have simplified to simply pulling the plug with "power off" from Hyper-V at the grub menu, do not close the Connection window, restart, and 19.10 will boot fast. Note that "system setup" (I previously mistakenly wrote "system startup") doesn't appear in the grub menu of stock 19.10; it appears only after updating packages in Ubuntu. Yes, I am able to select the “restart” button with the keyboard. But then the shortcut doesn’t work; what triggers the shortcut is powering off the VM.

https://chaetura.net/ms-vid2-bug-1848534.webm

Furthermore, the shortcut boot of 19.10 solves *THREE* significant problems with stock 19.10 that are not problems with stock 18.04.3. Reasonable conclusion is that all three problems have the same cause, which somehow the shortcut boot avoids:

1) avoids the long delay in startup

2) allows user to select any screen size for the Connection (whereas stock 19.10 came in one size only and user has no opportunity to select; goes hand in hand with the different login screen, as you can see in video.

3) after login, cut-and-paste from guest to host is possible. Not functional with stock 19.10.

I see that graphical artifact at some point during boot of any Ubuntu VM in Hyper-V. You can see it in the video too. In 19.10 it comes while the Spectre mitigation message is on the screen in console, at a point where console changes its rendering/mode somehow and the message reappears in a slightly different font.

I forgot to show /proc/cmdline and "uname -a" in the video but for 19.10 it is:

   BOOT_IMAGE=/boot/vmlinuz-5.3.0-23-generic root=PARTUUID=[blahblah] ro quiet splash
   Linux stock19 5.3.0-23-generic #25-Ubuntu SMP Tue Nov 12 09:22:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

and in 18.04.3 it is

    BOOT_IMAGE=/boot/vmlinuz-5.0.0-25-generic root=LABEL=desktop-rootfs ro quiet splash vt.handoff=1
    Linux stock18 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Note that I installed my 19.10 via Hyper-V Manager's "quick-create" rather than from an .iso downloaded separately. Same kernel version, but mine is an earlier packaging number, and who knows what else may be different about the install image. The manager says the image was updated 17 October 2019. The Manager downloads a zip file from partner-images.canonical.com via https so I cannot snoop the URL looking at the packets. I have a copy of the .vhdx which is insie the zip file, and I'm messing around with trying to mount it in another VM but I haven't solved that yet.

My Windows 10 Pro x64 build (today) is Win10: Version 1909 (OS Build 18363.778). No further windows updates are available, and I don't understand Windows version numbers, so I can't explain the discrepancy from yours.

The startup delay persists after updating the stock install of 19.10 (using aptitude; all available updates accepted b...

Read more...

Revision history for this message
Matthew Swift (msgallery) wrote :
Download full text (3.5 KiB)

Now we are getting somewhere.

X windows system implicated. Though the 2d video shows how fast the shortcut boot is, I thought I should get a firm number so I did the shortcut boot and logged in to get output of 'systemd-analyze critical-chain' and I got the following very interesting responses, shell dialogue copied below. The same processes (I would assume) are hanging for 1m44s, but the shortcut somehow allows the user to log in while waiting for them to finish (as well as solving the other problems of clipboard and screen size, which seem to go with the more primitive login screen). Whether those processes should take 1m44s or not, I don't know, maybe it is normal, but the user should be able to log in before they are complete, as I can do with shortcut boot. If those processes are taking inordinately long, then shortcut boot gives opportunity to log in and catch them as they struggle, as seen below. The time I had to wait to get the critical-chain result was about as long as I had to wait to log in at all, under stock 19.10 without shortcut-boot procedure:

wift@stock19:~$ sudo -i
[sudo] password for swift:
root@stock19:~# systemd-analyze critical-chain
Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).
Please try again later.
Hint: Use 'systemctl list-jobs' to see active jobs
root@stock19:~# systemctl list-jobs
JOB UNIT TYPE STATE
 48 setvtrgb.service start waiting
137 system-getty.slice start waiting
  1 graphical.target start waiting
102 systemd-update-utmp-runlevel.service start waiting
 83 plymouth-quit-wait.service start running
  2 multi-user.target start waiting

6 jobs listed.
root@stock19:~# systemd-analyze critical-chain
Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).
Please try again later.
Hint: Use 'systemctl list-jobs' to see active jobs
root@stock19:~# systemd-analyze critical-chain
Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).
Please try again later.
Hint: Use 'systemctl list-jobs' to see active jobs
root@stock19:~# systemd-analyze critical-chain
Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).
Please try again later.
Hint: Use 'systemctl list-jobs' to see active jobs
root@stock19:~# date
Sun 19 Apr 2020 10:50:27 PM EDT
root@stock19:~# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @1min 44.881s
└─multi-user.target @1min 44.881s
  └─xrdp.service @1.496s +1.036s
    └─xrdp-sesman.service @1.474s +20ms
      └─network.target @1.470s
        └─NetworkManager.service @1.381s +88ms
          └─dbus.service @1.379s
            └─basic.target @1.375s
              └─sockets.target @1.375s
                └─snapd.socket @1.373s +1ms
                  └─sysinit.target @1.372s
                    └─systemd-timesyncd.service @1.175s +196ms
                      └─systemd-tmpfiles-setup.service @1.160s...

Read more...

Revision history for this message
Dexuan Cui (decui) wrote :

I created a Ubuntu 19.10 VM via "Quick Create..." and still can not reproduce the long delay of > 1 minute: the VM can boot up to the Xorg GUI desktop in 26 seconds.

My Windows 10 has the same version info: Version 1909 (OS Build 18363.778).

At the grub screen, can you press 'e' and, manually edit the kernel parameter: please remove the "quiet splash $vt_handoff" and add "ignore_loglevel sysrq_always_enabled". You may want to enable the serial console logging by adding the kernel parameter "console=ttyS0", and attach putty (Run as Administrator) to the named pipe \\.\pipe\debug_slow_vm, assuming you configure the VM serial console by "Set-VMComPort -VMName your_vm_name -number 1 -path \\.\pipe\debug_slow_vm").

This way, you should get more messages on the VM serial console when the VM boots up. When you see the long delay, you can press SysRQ+w (i.e. the Right Alt + SysRq + w) to show the blocked processes, if any. This may provide more info about the long delay. BTW, here I assume your have a keyboard that has the SysRq key. :-)

It looks systemd can be configured to use "--log-level=debug --default-standard-output=kmsg --default-standard-error=kmsg", which may provide more info as well, if we check 'dmesg' and/or the VM serial console.

Revision history for this message
Dexuan Cui (decui) wrote :

It looks #48 shows some service is causing the long delay -- can you try 'systemctl list-jobs' to see active jobs, as the "Hint" says? :-)

Revision history for this message
Matthew Swift (msgallery) wrote :

You missed that I did include the output of 'systemctl list-jobs' during the crucial interval of delay.

I will follow your instructions regarding boot parameters etc. and post results asap.

I expect there is an enormous difference between accessing the VM via RPD (XRDP) protocol and accessing the VM via the more native interface (purple screen) which is eventually available on 19.10, even though both are accessed through the Hyper-V "Connection" interface. XRDP seems like the preferred way, it's the only way I get to 18.04, and forcing it on 19.10 is the only way I get screen size control and cut-and-paste from guest to host. Am seeking confirmation of what is the correct/preferred behavior is with 19.10. Should I be getting the "native" purple login screen quickly and then have cut-paste and screen control, or should be getting the XRDP interface quickly, just like I do with 18.04?

Revision history for this message
Matthew Swift (msgallery) wrote :

To clarify, I say the purple login screen is "more native" because that's what I get on the monitor of a physically independent machine (not a VM) running Ubunutu; naturally I do not get an (X)RDP screen on that machine ever.

Revision history for this message
Dexuan Cui (decui) wrote :

Sorry, I did miss this part of your previous reply:

root@stock19:~# systemctl list-jobs
JOB UNIT TYPE STATE
 48 setvtrgb.service start waiting
137 system-getty.slice start waiting
  1 graphical.target start waiting
102 systemd-update-utmp-runlevel.service start waiting
 83 plymouth-quit-wait.service start running
  2 multi-user.target start waiting

I'm wondering if you can disable setvtrgb.service, system-getty.slice, systemd-update-utmp-runlevel.service, and plymouth-quit-wait.service, and see if the long delay will disappear. I guess these 4 services don't look critical to the GUI desktop.

Revision history for this message
Dexuan Cui (decui) wrote :

I also tried xrdp mode and the VM booted up to the xrdp login window in 14 seconds, which is faster than the "native Xorg GUI mode" (which needs 30s)

Revision history for this message
Matthew Swift (msgallery) wrote : RE: [Bug 1848534] Re: [Microsoft Hyper-V guest] System shows graphic artifacts for a moment, then text cursor for about minute and then starts
Download full text (3.4 KiB)

I followed your instructions of 4/21 regarding changing kernel parameters and attaching PuTTY etc. Screenshot of the edited parameters is next to your email attached (if attachment won't get published, I will post online). I can send the PuTTY output, but I don't think we learned anything we didn't already see in dmesg. Console login prompt comes up quickly, then the long delay in the Hyper-V Connection window before arriving at the Xorg GUI login. Alt-SysRq-w during this time gives nothing, just two lines like these:

[ 70.428525] sysrq: Show Blocked State
[ 70.429990] task PC stack pid father

While doing this, I noticed today that some previously reliable ways to get to the "shortcut boot" (the XRDP login screen, which comes up quickly, instead of waiting for the Xorg screen) were no longer reliable (e.g., sometimes "power off" force in Hyper-V Manager and restart led to a long boot). I also discovered a state in which starting the VM from an "off" state in Hyper-V Manager apparently skipped even the Grub screen and went quickly to XRDP login screen. At this point I rebooted the host. Some deep kind of Windows caching going on? After reboot, behaviors earlier described became predictable again.

Three questions:

1) You wrote on 4/22 (second mail): "I also tried xrdp mode and the VM booted up to the xrdp login window in 14 seconds, which is faster than the "native Xorg GUI mode" (which needs 30s)". How did you intentionally "try XRDP mode"? How do you have any control over whether you get the XRDP or Xorg login screen? It seems to me I do not have any control. From a user's perspective, that is the whole problem here, how to get the XRDP login screen on first startup (and all subsequent) of a 19.10 VM within a given Hyper-V Connection window. As shown earlier, when the user gets the XRDP login, the (presumably) same processes are still being delayed for the same length of time as when the user is forced to wait for the Xorg login screen, but the user has meanwhile been able to log in via XRDP and begin doing his or her work. Furthermore, screen size control and cut-and-paste from guest to host are available only with the XRDP login. From user's perspective, I think the problem is solved once user can log into the VM and start working without waiting for a 90s timeout, and screen size control and cut-paste are operational. Which in my case seems to be equivalent to being able to get the XRDP login screen reliably; if I get it, it always arrives quickly.

2) You wrote on 4/21: "It looks systemd can be configured to use "--log-level=debug --default- standard-output=kmsg --default-standard-error=kmsg". Please advise me how to do this. I administered Debian systems from Buzz (1996) to Squeeze (2011), and a lot of the boot process is new to me. (In those days, one had to use Ctrl-S / Ctrl-Q to be able to read booting output.)

3) You wrote on 4/22: "I'm wondering if you can disable setvtrgb.service, system-getty.slice, systemd-update-utmp-runlevel.service, and plymouth-quit-wait.service, and see if the long delay will disappear. I guess these 4 services don't look critical to the GUI desktop." Likewise, please ...

Read more...

Revision history for this message
Dexuan Cui (decui) wrote :
Download full text (3.3 KiB)

Since Alt-SysRq-w gives nothing, I'm sure the long delay is not a kernel/driver issue but a user space issue. It looks due to some reason I just can not reproduce the long delay. :-(

In the Hyper-V Virtual Machine Connection window's "View" menu, there is an item "Enhanced Session". In my Ubuntu 19.10 VM created by "Quick Create...", the xrdp daemon/service is configured to automatically run during the boot-up procedure; I think as soon as the xrdp daemon starts to run, the "Enhanced Session" item becomes clickable/usable, and I can click it to toggle between "Enhanced Session" mode (i.e. xrdp mode) and the native Xorg GUI mode; when I'm in the xrdp mode, I click VM Connection window's Action | Shut Down, then Start, and the VM will boot up to the xrdp login screen in about 14 seconds; when I'm in the Xorg mode, I click Shutdown then Start, the VM will boot up to the Xorg GUI desktop in about 30 seconds. If I shut down the VM, close the VM Connection window, and start the VM and open VM Connection window, I'll be prompted by a small pop-up window to choose a resolution when (I think) the xrdp daemon starts to run: 1) if I click the close icon of the small window, I'll be in the Xorg GUI mode; if I accept the default resolution (or change to a different resolution) and click "connect" in the small window, I'll be in the xrdp mode. So all these work pretty good for me.

Note: after I just created the 19.10 VM by "Quick Create..." and set up the host name and user name/password stuff, I rebooted the VM and when the VM booted up, I found the "Enhanced Session" was not clickable/usable -- this looks like a bug -- while I still don't know the root cause, it looks this can be resolved by manually adding the line "initrd /boot/initrd.img-5.3.0-23-generic #This line is added by Dexuan manually" into "/boot/grub/grub.cfg":

menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-55829715-0091-4b86-b060-1cb88f342faf' {
...
        if [ "${initrdfail}" = 1 ]; then
          linux /boot/vmlinuz-5.3.0-23-generic root=PARTUUID=43e99d31-1277-402c-a13b-6cc8fb93169b ro quiet splash $vt_handoff
          initrd /boot/initrd.img-5.3.0-23-generic
        else
          linux /boot/vmlinuz-5.3.0-23-generic root=PARTUUID=43e99d31-1277-402c-a13b-6cc8fb93169b ro quiet splash $vt_handoff panic=-1
          initrd /boot/initrd.img-5.3.0-23-generic #This line is added by Dexuan manually!!!
        fi
        initrdfail
}

With the addition of the line, it looks "GRUB_TIMEOUT=0" in "/etc/default/grub" is always really applied every time I reboot the VM.
Without the line, it looks sometimes the grub timeout is 30 second and sometimes it's 0 second.
BTW, I reported a bug for the missing initrd line a few weeks ago for a different issue: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1870189.

I suggest you also manually add the line, then I guess you should be able to reliably toggle between xrdp mode and Xorg mode.

Note: "/boot/grub/grub.cfg" is overwritten when update-grub is run by us or some automatic-update daemon, so we may want to check if the line is still there when we see s...

Read more...

Revision history for this message
Dexuan Cui (decui) wrote :

I don't have much knowledge bout systemd, either :-) I just did a "man systemd" and found the options of systemd. "man systemd" says that we can use pass these kernel parameters to systemd:

systemd.service_watchdogs=true systemd.show_status=true systemd.log_level=debug systemd.dsystemd.default_standard_output=kmsg systemd.default_standard_error=kmsg

I tried these by adding them into /boot/grub/grub.cfg manually, at the end of the line "linux /boot/vmlinuz-5.3.0-23-generic ...".
I also replaced "quiet splash $vt_handoff" with "ignore_loglevel". So I can get more messages from systemd, but not so much as I expected. Not sure if this would be helpful to troubleshoot the long delay issue for you, and I'm not even sure if I enabled the systemd loggong completely correctly -- again, I'm not really familiar with systemd. :-)

To stop/disble a systemd "service", I think we can use something like this (taking the setvtrgb.service as an example):
  systemctl stop setvtrgb.service
  systemctl disable setvtrgb.service
  systemctl status setvtrgb.service

Revision history for this message
Dexuan Cui (decui) wrote :

Sorry, I made a typo above: systemd.dsystemd.default_standard_output=kmsg ==> systemd.default_standard_output=kmsg.
BTW, it looks systemd.show_status=true makes no difference for me. I don't see any status info during the boot-up time -- not sure if I did something wrong.

Revision history for this message
M (xh-parcen-et) wrote :

Honestly speaking two notes:

1. forget Ubuntu 19.10 -> there is LTS 20.04

2. after ca. 5 or 6 months Microsoft has not updated firmware for my CPU in standard Windows Update packages and because of it I'm still not interested in using HyperV (it's interfering with driver updating microcode for me) -> Sorry

Revision history for this message
Dexuan Cui (decui) wrote :

Thanks for the reminder! I just realized Ubuntu 20.02 was already released on 4/23. We should try it.

For the CPU firmware (CPU microcode?) update issue: sorry, it's completely out of my scope -- I only work on Linux. Hopefully that issue will be resolved in the near future.

Revision history for this message
Scott Beamer (mr-scott-beamer) wrote :

This is still a bug in Ubuntu 21.04 and 21.10.

To post a comment you must log in.