VNC unusable when booting ubuntu-20.04.1-live-server-amd64.iso

[EOLE team]

[Impact]

 * Upstream [1] added a mistake missing an else downgrading
   cirrus to vga.

 * Fixed in [2] and released in 1.14 but needs to be backported to Focal.

[1]: https://github.com/coreboot/seabios/commit/34b6ecc160749a691b80fcb8638216518d971c65
[2]: https://github.com/coreboot/seabios/commit/d11c75185276ded944f2ea0277532b7fee849bbc

[Test Case]

* Get an Ubuntu CD image and boot it with Cirrus video.

$ wget https://mirrors.ocf.berkeley.edu/ubuntu-releases/20.04/ubuntu-20.04.1-live-server-amd64.iso
$ qemu-system-x86_64 -m 2048 -smp 1 -vnc 0.0.0.0:1 -k fr -name test-boot -monitor pty -boot order=d -drive file=/var/lib/libvirt/images/ubuntu-20.04.1-live-server-amd64.iso,media=cdrom -device cirrus-vga,id=video0,bus=pci.0,addr=0x3

Once that is running you can connect VNC to :5901 and with the error will see a fail loop stating:
  graphics initialization failed
  Error setting up gfxboot
  boot:
  Could not allocate memory.

With the fix this will boot into the installer of the ISO.

[Where problems could occur]

 * This changes the listing of (virtual) GPUs to the guest.
   There it differs between Cirrus/VGA/stdvga/bochs.
   Therefore using those backends are the ones we'd expect errors.
   On testing no issue was seen thou, and the code is in Ubuntu since
   groovy without breaking the world all around.

[Other Info]

 * Depending where/how you drive your VMs the actual bug might be well
   hidden and it just appears as a hanging or not starting boot.

----

Hello.

I can't boot ubuntu-20.04.1-live-server-amd64.iso with VNC graphic
using libvirt, there is an error message:

  graphics initialization failed
  Error setting up gfxboot
  boot:

after a moment, I see some other messages:

  graphics initialization failed
  Error setting up gfxboot
  boot:
  Could not allocate memory.
  boot
  Could not allocate memory.
  boot
  Could not allocate memory.
  boot

To reproduce:

- download ubuntu-20.04.1-live-server-amd64.iso to
  /var/lib/libvirt/images

- create a qcow2 image /var/lib/libvirt/images/disk0.qcow2

- create the following libvirt XML definition file

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 <name>ubuntu-server-20.04.1</name>
 <title>ubuntu-server-20.04.1</title>
 <vcpu><![CDATA[4]]></vcpu>
 <cputune>
  <shares>205</shares>
 </cputune>
 <memory>2097152</memory>
 <os>
  <type arch='x86_64'>hvm</type>
 </os>
 <devices>
  <emulator><![CDATA[/usr/bin/qemu-system-x86_64]]></emulator>
  <disk type='file' device='disk'>
   <source file='/var/lib/libvirt/images/disk0.qcow2'/>
   <target dev='sda'/>
   <boot order='1'/>
   <driver name='qemu' type='qcow2' cache='none'/>
   <address type='drive' controller='0' bus='0' target='0' unit='0'/>
  </disk>
  <disk type='file' device='cdrom'>
   <source file='/var/lib/libvirt/images/ubuntu-20.04.1-live-server-amd64.iso'/>
   <target dev='hda' bus='ide'/>
   <boot order='2'/>
   <readonly/>
   <driver name='qemu' type='raw' cache='none'/>
  </disk>
  <graphics type='vnc' listen='0.0.0.0' port='5901' keymap='fr'/>
  <input type='tablet' bus='usb'/>
 </devices>
 <features>
  <acpi/>
 </features>
</domain>

- define the domaine with: virsh define ubuntu-server.xml

- start the domain with: virsh start ubuntu-server-20.04.1

- connect to the VNC port 5901

It's working fine when I try the following:

  qemu-system-x86_64 -m 2048 -smp 1 -vnc 0.0.0.0:1 -k fr -name test-boot -monitor pty -boot order=d -drive file=/var/lib/libvirt/images/disk0.qcow2,media=disk -drive file=/var/lib/libvirt/images/ubuntu-20.04.1-live-server-amd64.iso,media=cdrom

The non working qemu command line used by libvirt is:

  /usr/bin/qemu-system-x86_64 -name guest=ubuntu-server-20.04.1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-7-ubuntu-server-20.04./master-key.aes -machine pc-i440fx-focal,accel=kvm,usb=off,dump-guest-core=off -cpu qemu64 -m 2048 -overcommit mem-lock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 503e911c-f7a5-48ef-8241-36bdbc75b9a3 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -no-acpi -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x3 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/disk0.qcow2","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device scsi-hd,bus=scsi0.0,scsi-id=0,device_id=drive-scsi0-0-0,drive=libvirt-2-format,id=scsi0-0-0,bootindex=1,write-cache=on -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/ubuntu-20.04.1-live-server-amd64.iso","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"} -device ide-cd,bus=ide.0,unit=0,drive=libvirt-1-format,id=ide0-0-0,bootindex=2,write-cache=on -vnc 0.0.0.0:1 -k fr -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.14
Architecture: amd64
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
Package: libvirt-daemon-system 6.0.0-0ubuntu8.5
PackageArchitecture: amd64
ProcCmdline: BOOT_IMAGE=/vmlinuz-5.4.0-58-generic root=/dev/mapper/vggrichka-root ro quiet splash vt.handoff=7
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=fr_FR.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 5.4.0-58.64-generic 5.4.73
Tags: focal
Uname: Linux 5.4.0-58-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.qemu-announce-self.xml: [modified]
modified.conffile..etc.libvirt.qemu.conf: [modified]
modified.conffile..etc.libvirt.qemu.networks.default.xml: [modified]
mtime.conffile..etc.libvirt.nwfilter.allow-arp.xml: 2020-12-17T10:31:59.619331
mtime.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: 2020-12-17T10:31:59.639331
mtime.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: 2020-12-17T10:31:59.599330
mtime.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: 2020-12-17T10:31:59.643331
mtime.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: 2020-12-17T10:31:59.575330
mtime.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: 2020-12-17T10:31:59.647331
mtime.conffile..etc.libvirt.nwfilter.clean-traffic.xml: 2020-12-17T10:31:59.631331
mtime.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: 2020-12-17T10:31:59.615331
mtime.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: 2020-12-17T10:31:59.591331
mtime.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: 2020-12-17T10:31:59.611331
mtime.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: 2020-12-17T10:31:59.595330
mtime.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: 2020-12-17T10:31:59.627331
mtime.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: 2020-12-17T10:31:59.603331
mtime.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: 2020-12-17T10:31:59.607331
mtime.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: 2020-12-17T10:31:59.583330
mtime.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: 2020-12-17T10:31:59.623331
mtime.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: 2020-12-17T10:31:59.579330
mtime.conffile..etc.libvirt.nwfilter.qemu-announce-self.xml: 2020-12-17T10:31:59.587330
mtime.conffile..etc.libvirt.qemu.conf: 2020-12-17T10:32:02.111360
mtime.conffile..etc.libvirt.qemu.networks.default.xml: 2020-12-17T10:31:59.179326

[EOLE team (eole-team)]

apport information

[EOLE team (eole-team)]

apport information

[EOLE team (eole-team)]

apport information

[EOLE team (eole-team)]

apport information

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[EOLE team (eole-team)]

I tested on hirsute and it's working fine.

I don't see meaningful differences between focal and hirsute:

- For the XML

--- ubuntu-server-20.04.1-on-focal.xml 2020-12-17 15:02:28.678279263 +0100
+++ ubuntu-server-20.04.1-on-hirsute.xml 2020-12-17 14:47:50.790210994 +0100
@@ -1,6 +1,6 @@
 <domain type='kvm' id='1'>
   <name>ubuntu-server-20.04.1</name>
- <uuid>65166cc6-7cec-47e3-bd85-390f60a71d01</uuid>
+ <uuid>48ce4792-c557-4e02-a946-a23d23cabf85</uuid>
   <title>ubuntu-server-20.04.1</title>
   <memory unit='KiB'>2097152</memory>
   <currentMemory unit='KiB'>2097152</currentMemory>
@@ -12,7 +12,7 @@
     <partition>/machine</partition>
   </resource>
   <os>
- <type arch='x86_64' machine='pc-i440fx-focal'>hvm</type>
+ <type arch='x86_64' machine='pc-i440fx-hirsute'>hvm</type>
   </os>
   <features>
     <acpi/>
@@ -88,8 +88,8 @@
     </memballoon>
   </devices>
   <seclabel type='dynamic' model='apparmor' relabel='yes'>
- <label>libvirt-65166cc6-7cec-47e3-bd85-390f60a71d01</label>
- <imagelabel>libvirt-65166cc6-7cec-47e3-bd85-390f60a71d01</imagelabel>
+ <label>libvirt-48ce4792-c557-4e02-a946-a23d23cabf85</label>
+ <imagelabel>libvirt-48ce4792-c557-4e02-a946-a23d23cabf85</imagelabel>
   </seclabel>
   <seclabel type='dynamic' model='dac' relabel='yes'>
     <label>+64055:+108</label>

- For the QEMU command lines

--- qemu-cmdline-focal.txt 2020-12-17 15:04:52.973631013 +0100
+++ qemu-cmdline-hirsute.txt 2020-12-17 14:46:24.018597093 +0100
@@ -2,12 +2,12 @@
 -name guest=ubuntu-server-20.04.1,debug-threads=on
 -S
 -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-ubuntu-server-20.04./master-key.aes
--machine pc-i440fx-focal,accel=kvm,usb=off,dump-guest-core=off
+-machine pc-i440fx-hirsute,accel=kvm,usb=off,dump-guest-core=off
 -cpu qemu64
 -m 2048
 -overcommit mem-lock=off
 -smp 4,sockets=4,cores=1,threads=1
--uuid 65166cc6-7cec-47e3-bd85-390f60a71d01
+-uuid 48ce4792-c557-4e02-a946-a23d23cabf85
 -no-user-config
 -nodefaults
 -chardev socket,id=charmonitor,fd=31,server,nowait

So, it may be a qemu issue?

[EOLE team (eole-team)]

I don't have issue to boot the debian-10.7.0-amd64-DVD-1.iso

[Christian Ehrhardt  (paelzer)]

Hi,
first of all thanks for your report!

Trying to recreate your issue I've downloaded the same ubuntu-20.04.1-live-server-amd64.iso to a focal and a hirsute host system.
Then I've defined the qcow2 and XML you posted on those.

$ wget https://mirrors.ocf.berkeley.edu/ubuntu-releases/20.04/ubuntu-20.04.1-live-server-amd64.iso
$ qemu-img create -f qcow2 /var/lib/libvirt/images/disk0.qcow2 10G
$ virsh define ubuntu-server.xml

Just as you reported and one would expect there s no difference between the two other than the UUID and the default machine type.

Both VMs start fine from a libvirt/qemu POV.

Since recently libvirt 6-9 and qemu 5.2 migrated my "new" case also has
-machine pc-i440fx-hirsute,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-object memory-backend-ram,id=pc.ram,size=2147483648 \

But still very similar to your report so far.

The important VNC and display config in both cases is:
-vnc 0.0.0.0:1 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \

Note: Cirrus is embarrassingly old, but is and will stay the default for compatibility. Never the less I'd recommend using QXL - at least for any Linux guests these days.

Connecting VNC to those two guests confirms your report.
The one running under Focal fails with "Could not allocate memory. boot" the one under Hirsute works fine.

[Christian Ehrhardt  (paelzer)]

As I've expected above - the modern default of QXL (that any higher level software usually would use) works fine.

I was trying other backends: vga, vmvga, virtio also all just work fine.

The reason why your stripped down qemu commandline works is because qemu has switched the defaults.
Since qemu 2.2 your default will be vga (called std for qemu). Therefore since you had no '-device cirrus-vga,...' your manual command worked for you.
If you append '-device cirrus-vga,id=video0,bus=pci.0,addr=0x3' it fails just the same.

Also the error seems to be at the bootloader or at least early-boot, not the later stages.
While there might be an issue in kernel or grub in the guest that is triggered here - it seems that some fix went into cirrus emulation to avoid that issue. So it can be fixed at the host level.

So the question is what is breaking the cirrus video device and what/how was it fixed later.

[Christian Ehrhardt  (paelzer)]

Via server backports [1] I was trying the version of Groovy "in place"
I expected the change to be in qemu, so I updated just the qemu components from 1:4.2-3ubuntu6.10 to 1:5.0-5ubuntu9.2~backport20.04-202011301725~ubuntu20.04.1.
Same error on 5.0 still.

I still had a qemu 5.1 for Hirsute at [2] that tried next.

Note: a praise to the neat co-installability of libs in Ubuntu/Debian - apt-cache policy was rather crowded now :-)

Trying qemu 1:5.1+dfsg-4ubuntu1~ppa4 still failed.
Trying qemu 1:5.2+dfsg-2ubuntu1 still failed as well.

All other components are still mostly on the level of Focal.
Ok, assumptions be damned - it seems to be something else than qemu.

Next best guess - the bios.
Trying seabios 1.14.0-2 + qemu 1:5.2+dfsg-2ubuntu1 then worked.

Downgrading qemu back to Focal at 1:4.2-3ubuntu6.10 but keeping seabios at 1.14.0-2 then still worked. 1.14.0-1~backport20.04-202010240636~ubuntu20.04.1 is only a minor change away and also working fine.
So the fix is solely in the seabios component it seems.

[1]: https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4321

[Christian Ehrhardt  (paelzer)]

For anyone affected there are a bunch of workarounds available now that we cornered the issue a bit.

a) do use a more modern default than cirrus (recommended anyway), QXL being the most common one
b) use the seabios package from https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports
c) upgrade to Ubuntu 20.10 or later

[Christian Ehrhardt  (paelzer)]

1.13 contained https://github.com/coreboot/seabios/commit/34b6ecc16074
Which was fixed by https://github.com/coreboot/seabios/commit/d11c751852 in 1.14

The patch is small and at least according to the description likely to be related.
No other commits 1.13 -> 1.14 mention cirrus.

I'll prep a PPA with Focals seabios package plus that fix for a test.

[Christian Ehrhardt  (paelzer)]

Building in Test PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4386/+packages
Packaging MP: https://code.launchpad.net/~paelzer/ubuntu/+source/seabios/+git/seabios/+merge/395769

[Christian Ehrhardt  (paelzer)]

Tested on the 1.13.0-1ubuntu1.1~ppa1 PPA build and working fine with that.
Opening up the MP for review to get a second pair of eyes on it.

[EOLE team (eole-team)]

Thanks a lot

> a) do use a more modern default than cirrus (recommended anyway), QXL being the most common one

Maybe libvirt should switch the default?

[Robie Basak (racb)]

Hello EOLE, or anyone else affected,

Accepted seabios into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/seabios/1.13.0-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

https://launchpad.net/ubuntu/+source/seabios/1.13.0-1ubuntu1.1 is built in Focal proposed now.

In an env that failed with 1.13.0-1ubuntu1
I was testing with 1.13.0-1ubuntu1.1 and it works fine.

Re-downgrading establishes the error again - so yeah the new build in proposed fixes the issue.

[EOLE team (eole-team)]

I confirm that it's working with version 1.13.0-1ubuntu1.1.

[Launchpad Janitor (janitor)]

This bug was fixed in the package seabios - 1.13.0-1ubuntu1.1

---------------
seabios (1.13.0-1ubuntu1.1) focal; urgency=medium

  * d/p/lp-1908512-vga-fix-cirrus-bios.patch: fix boot with cirrus
    (LP: #1908512)

 -- Christian Ehrhardt <email address hidden> Tue, 05 Jan 2021 09:52:28 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for seabios has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.