[regression] vgabios -> seabios breaks (my) 16-bit applications

Bug #1404396 reported by Richard Laager on 2014-12-19
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
seabios (Ubuntu)
High
Unassigned
Trusty
High
Unassigned

Bug Description

=======================================================
Impact: Unable to run 16-bit applications in windows
Fix: turn off VGA_ALLOCATE_EXTRA_STACK
Test case: (see comment #21)
Regression potential: This stops the allocation of extra vga stack so could
 conceivably break other applications. However all tests cases in
 the qemu regression tests pass, and this has been enabled for some
 time in wily.
=======================================================
Upgrading from Ubuntu Precise to Ubuntu Trusty broke one of my guests.

I've recently upgraded a host from Ubuntu Precise (qemu-kvm-1.0) to Ubuntu Trusty (qemu 2.0.0). I have a Windows 2008 32-bit OS running on a 64-bit VM that runs a 16-bit line of business application. (While I realize that "upgrade the application" is the right answer, I can't.) The Windows VM boots, but the business application crashes on startup. Changing the VM to be a 32-bit VM doesn't help.

After several hours of compiling and testing intermediate qemu versions, I accidentally stumbled into the real issue. Trusty has switched from vgabios to seabios:

PRECISE $ ls -la /usr/share/qemu/vgabios*
lrwxrwxrwx 1 root root 22 Dec 10 18:37 /usr/share/qemu/vgabios.bin -> ../vgabios/vgabios.bin
lrwxrwxrwx 1 root root 29 Dec 10 18:37 /usr/share/qemu/vgabios-cirrus.bin -> ../vgabios/vgabios.cirrus.bin
lrwxrwxrwx 1 root root 26 Dec 10 18:37 /usr/share/qemu/vgabios-qxl.bin -> ../vgabios/vgabios.qxl.bin
lrwxrwxrwx 1 root root 29 Dec 10 18:37 /usr/share/qemu/vgabios-stdvga.bin -> ../vgabios/vgabios.stdvga.bin
lrwxrwxrwx 1 root root 29 Dec 10 18:37 /usr/share/qemu/vgabios-vmware.bin -> ../vgabios/vgabios.vmware.bin

TRUSTY $ ls -la /usr/share/qemu/vgabios*
lrwxrwxrwx 1 root root 29 Dec 10 18:50 /usr/share/qemu/vgabios.bin -> ../seabios/vgabios-isavga.bin
lrwxrwxrwx 1 root root 29 Dec 10 18:50 /usr/share/qemu/vgabios-cirrus.bin -> ../seabios/vgabios-cirrus.bin
lrwxrwxrwx 1 root root 26 Dec 10 18:50 /usr/share/qemu/vgabios-qxl.bin -> ../seabios/vgabios-qxl.bin
lrwxrwxrwx 1 root root 29 Dec 10 18:50 /usr/share/qemu/vgabios-stdvga.bin -> ../seabios/vgabios-stdvga.bin
lrwxrwxrwx 1 root root 29 Dec 10 18:50 /usr/share/qemu/vgabios-vmware.bin -> ../seabios/vgabios-vmware.bin

I installed the vgabios package and made my own tree of symlinks:

$ ls -l /usr/share/wiktel-server-vmhost/vgabios/
total 0
lrwxrwxrwx 1 root root 27 Dec 17 00:40 bios.bin -> /usr/share/seabios/bios.bin
lrwxrwxrwx 1 root root 23 Dec 17 00:40 keymaps -> /usr/share/qemu/keymaps
lrwxrwxrwx 1 root root 30 Dec 17 00:40 vgabios.bin -> /usr/share/vgabios/vgabios.bin
lrwxrwxrwx 1 root root 37 Dec 17 00:40 vgabios-cirrus.bin -> /usr/share/vgabios/vgabios.cirrus.bin
lrwxrwxrwx 1 root root 34 Dec 17 00:40 vgabios-qxl.bin -> /usr/share/vgabios/vgabios.qxl.bin
lrwxrwxrwx 1 root root 37 Dec 17 00:40 vgabios-stdvga.bin -> /usr/share/vgabios/vgabios.stdvga.bin
lrwxrwxrwx 1 root root 37 Dec 17 00:40 vgabios-vmware.bin -> /usr/share/vgabios/vgabios.vmware.bin

Then I added this to the guest XML:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <qemu:commandline>
    <qemu:arg value='-L'/>
    <qemu:arg value='/usr/share/wiktel-server-vmhost/vgabios'/>
  </qemu:commandline>
</domain>

This makes it work. Is this the long-term answer? Or should this change from vgabios to seabios be reverted? Or should libvirt be modified to accept <rom file="..."> for <video>, as discussed here: https://bugzilla.redhat.com/show_bug.cgi?id=811227#c9 Or should I try to get the seabios developers to help debug and fix this?

If this could be fixed in seabios that would be ideal. Discussion of a
second-best solution should probably wait unti lwe're certain that is
not possible.

Since with respect to this issue we should be in sync with Debian, I'm curious
whether Debian also has this issue.

Serge Hallyn (serge-hallyn) wrote :

There have been a few updates to seabios for better handling legacy
apps, i wonder if one of them might fix this bug.

Richard Laager (rlaager) wrote :

I built an updated seabios using the source from git and the debian directory from Debian's 1.7.5-1. It has the same problem.

Richard Laager (rlaager) wrote :

I rebuilt seabios git with CONFIG_DEBUG_LEVEL=8 and added the following options, as per: http://www.seabios.org/pipermail/seabios/2011-May/001718.html

  <qemu:commandline>
    <qemu:arg value='-chardev'/>
    <qemu:arg value='stdio,id=seabios'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/>
  </qemu:commandline>

I've attached the log file from the guest. All of the log output is from booting. No additional output appears when running the 16-bit application. I've also attached screenshots of the error in the guest, though I doubt they'll be very useful.

Richard Laager (rlaager) wrote :
Richard Laager (rlaager) wrote :
Paolo Bonzini (bonzini) wrote :

Can you reproduce with other 16-bit applications (e.g. skifree from http://ski.ihoc.net/#download)?

Paolo Bonzini (bonzini) wrote :

Also, can you attach a screenshot of the 16-bit application with vgabios, so we can have a clue of why the vgabios is involved?

Richard Laager (rlaager) wrote :

Yes, this is reproducible with 16-bit Skifree. There's more discussion on the mailing list, but the takeaway so far is that I can confirm that setting CONFIG_VGA_ALLOCATE_EXTRA_STACK=n fixes this.

Serge Hallyn (serge-hallyn) wrote :

Thanks - just to be clear, was ther eany downside to setting CONFIG_VGA_ALLOCATE_EXTRA_STACK=n?

Changed in qemu (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
importance: Medium → High
Richard Laager (rlaager) wrote :

There's no downside for me. http://www.seabios.org/pipermail/seabios/2015-January/008546.html and http://www.seabios.org/pipermail/seabios/2015-January/008547.html have a couple of answers. Basically, CONFIG_VGA_ALLOCATE_EXTRA_STACK improves things for some old DOS programs. No examples were given.

Richard Laager (rlaager) wrote :

I see I haven't given a status update here... There were a couple of patches on the mailing list. One definitely fixes it. The other fixes it, but possibly (I should retest) breaks video output while booting (on seabios git, but not 1.7.4).

Serge Hallyn (serge-hallyn) wrote :

So is the patch in http://www.seabios.org/pipermail/seabios/2015-January/008546.html the one which definately fixes it?

Richard Laager (rlaager) wrote :

No, that's the patch which adds EXTRA_STACK and breaks it.

Kevin O'Connor intends to finish up a patch using his option 5 soon:
> I'm inclined to go with option 5. The real patch for that option,
> however, is a little bigger. I should be able to send a more complete
> patch in the next couple of days.
-- http://www.seabios.org/pipermail/seabios/2015-January/008562.html

There's already a patch he wrote which fixes this using a different approach, but if he's going to fix this upstream shortly, we might as well wait a few more days for that.

Richard Laager (rlaager) wrote :

Upstream has a new patch that fixes this without any known breakage. I tested it and confirmed it fixes things for me. However, it doesn't apply to 1.7.4. I propose setting CONFIG_VGA_ALLOCATE_EXTRA_STACK=n until the next seabios release after 1.8.1.

Changed in seabios (Ubuntu):
importance: Undecided → High
status: New → Triaged
no longer affects: qemu (Ubuntu Trusty)
no longer affects: qemu (Ubuntu)
Serge Hallyn (serge-hallyn) wrote :

Looks like the simplest way to do it is to set the default in vgasrc/Kconfig.

Unfortunately it's late in the cycle to put this into 15.04. So for 15.10 we will hopefully take the full upstream fix, and do the vga-allocate-extra-stack=n fix for the SRUs. But the devel release must be fixed before we can SRU.

Could you please test the package I uploaded to

http://people.canonical.com/~serge/seabios_1.7.4-4ubuntu0.1_all.deb

?

Richard Laager (rlaager) wrote :

I can confirm that package fixes my applications (as well as SkiFree) on Ubuntu Trusty.

Serge Hallyn (serge-hallyn) wrote :

So given comment #16, the 1.8.1 package currently in wily would still need htat change?

If not, I'll mark it fix released. if so, I'll push a new package with that change and then push the proposed trusty package.

Changed in seabios (Ubuntu Trusty):
importance: Undecided → High
Serge Hallyn (serge-hallyn) wrote :

Also, for the SRU process, could you add a concise test case to the description above?

Richard Laager (rlaager) wrote :

Try to run 16-bit SkiFree in a 32-bit Windows OS inside a VM. You can download it here: http://ski.ihoc.net/ski.zip It does not work on Trusty (presumably through Wily) as-is, but if you set CONFIG_VGA_ALLOCATE_EXTRA_STACK=n, it will work. (I haven't re-tested this today, but that's what I and upstream were finding.)

Serge Hallyn (serge-hallyn) wrote :

On a stock wily kvm instance with 32-bit win8.1, ski.zip runs fine. I'll assume that seabios 1.8.1-2ubuntu1 fixed it, and proceed to testing the CONFIG_VGA_ALLOCATE_EXTRA_STACK=n in prep for final SRU to trusty.

Changed in seabios (Ubuntu):
status: Triaged → Fix Released
Serge Hallyn (serge-hallyn) wrote :

Hm, I installed win8.1 32-bit in kvm inside a trusty container, but still ski.zip runs fine for me.

Do I need to run an older windows release? Is there some amount of time I should play before ski.zip crashes?

Richard Laager (rlaager) wrote :

I realized later that my laptop doesn't run Trusty. It runs vivid. So disregard my previous comment.

Anyway, we're running Windows Server 2008 (not R2, which is only available in 64-bit). That's based on Vista, so maybe try Vista? This is still reproducible for me in my test environment. SkiFree 16-bit crashes immediately on startup. I never even see the game window.

I'd be happy to test any packages you build for the SRU.

Serge Hallyn (serge-hallyn) wrote :

Thanks, in vista i was able to reproduce. Just turning the default in vga/Kconfig to n fixes it. I also tried debian's 1.8.2 package but it fails.

I'm going to run full qemu and libvirt testsuites (from lp;qa-regression-testing) with that package to hopefully find any regressions, then merge debian's 1.8.2 and add this patch on top.

Richard Laager (rlaager) wrote :

Excellent!

description: updated
Serge Hallyn (serge-hallyn) wrote :

Actually, I just got a failure from the qa regression tests:

======================================================================
FAIL: test_virtio_net (__main__.QemuGuestVirtio)
Test virtio net
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test-qemu.py", line 1441, in test_virtio_net
    self._ssh_cmd(['lsmod'], search="virtio_net")
  File "./test-qemu.py", line 1203, in _ssh_cmd
    self.assertEquals(expected, rc, result + report)
AssertionError: Got exit code 255, expected 0
Connection timed out during banner exchange

Serge Hallyn (serge-hallyn) wrote :

fwiw I've not been able to reproduce this failure.

Hello Richard, or anyone else affected,

Accepted seabios into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/seabios/1.7.4-4ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in seabios (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Richard Laager (rlaager) wrote :

I will test the proposed package. I'm leaving for the weekend, so it won't be until Sunday night / Monday morning (US/Central) at best, and possibly the next night. But I will report back ASAP.

Simon Déziel (sdeziel) wrote :

Hi Richard, were you able to test the proposed package? Thanks

Richard Laager (rlaager) wrote :

I can confirm that the proposed package fixes the issue. I tested before the change by rebooting the guest using the stock SeaBIOS, and it was broken. I installed seabios-1.7.4-4ubuntu1 from trusty-proposed and rebooted the guest. It works now.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package seabios - 1.7.4-4ubuntu1

---------------
seabios (1.7.4-4ubuntu1) trusty-proposed; urgency=medium

  * Turn off VGA_ALLOCATE_EXTRA_STACK (LP: #1404396)

 -- Serge Hallyn <email address hidden> Wed, 26 Aug 2015 15:25:11 -0500

Changed in seabios (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for seabios has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This is all of such a long time ago, I've read through the old Mail threads and seabios git.
As far as I can tell this would still be an issue.

I'm a bit concerned we carry this Delta along like forever.
To have some hope resolving that I wanted to ask you @Richard if you could still retest it with todays versions. Like on Bionic with seabios >=1.10 and a recent qemu.

I could also build you a ppa with the 1.11.1 versions if you want - I'd just need someone that is sure how to test this 16bit UI app issues from the experience on it in the past.
So if you think you could re-test this - please let me know.

Richard Laager (rlaager) wrote :

Here's what I propose as a test plan:

1) I build seabios 1.10.2-1ubuntu1 (from bionic) on xenial. I install it, reboot the guest, and verify things still work.
2) I remove debian/patches/turn-vga-allocate-extra-stack-off, rebuild, and repeat the above test.

If that works, you should be able to drop the patch for cosmic and forward (or SRU to remove it in bionic, if you can get that SRU approved).

I think we are fine doing that change >=Cosmic if we can confirm it is working now. We have no regression that would make us need a SRU, this is more a request to simplify maintenance and get in sync with upstreams defaults again if they are ok now.

I really appreciate your help and your self-reliance on this - if you need me for anything let me know. Otherwise I'll just stay subscribed and wait for your results.

Thanks in advance!

Richard Laager (rlaager) wrote :

I performed the above testing, and it works.

Thank you so much Richard!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers