Regression: "Undefined video mode" with 3.5.0-14.16

[Jeremy Bicha]

Starting last night, Ubuntu images started showing an error message after the isolinux "press any key" screen but before plymouth. This is affecting today's daily images. Screenshot attached.

"Undefined video mode number: 0
Press <ENTER> to see video modes available, <SPACE> to continue, or wait 30 sec"

cjwatson said that this was related to the kernel:
arch/x86/boot/video.c:332: printf("Undefined video mode number: %x\n", mode);

I don't believe this affects installed Ubuntu with grub2. The Ubuntu install images use isolinux instead.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: linux-image-3.5.0-14-generic 3.5.0-14.16
ProcVersionSignature: Ubuntu 3.5.0-14.15-generic 3.5.3
Uname: Linux 3.5.0-14-generic x86_64
ApportVersion: 2.5.1-0ubuntu7
Architecture: amd64
Date: Wed Sep 12 08:28:01 20125
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-14-generic root=UUID=da074346-bf87-4cf5-a9ec-8589cb501c4e ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.5.0-14-generic N/A
 linux-backports-modules-3.5.0-14-generic N/A
 linux-firmware 1.91
SourcePackage: linux

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.6 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-rc5-quantal/

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1049650

[Seth Forshee (sforshee)]

This message comes from early boot code, which hasn't changed recently. The kernel includes a few options for handling video in early boot. Each option defines a struct that gets linked into a section named .videocards, and the kernel iterates through the structs in this section looking for any which can provide mode information. Somehow we're ending up with nothing in that section.

$ readelf -s setup.elf | grep video_card
   138: 00003660 0 NOTYPE GLOBAL DEFAULT 12 video_cards
   151: 00003660 0 NOTYPE GLOBAL DEFAULT 12 video_cards_end

We should see some space between video_cards and video_cards_end, and the fact that we don't means the kernel isn't going to find any options for video.

I verified that beta 1 (3.5.0-13.14) doesn't have this issue. None of the kernel changes between these versions look like they should have any impact, so my best guess is that this is caused by some change to the build tools. Those kernels are built with the same gcc version, but it does looke like binutils changed recently.

[Tim Lunn (darkxst)]

3.5.0-14.15 is unaffected also.

I tried testing the mainline kernel, but couldnt get it to install in the livecd chroot, grub complains about missing / device.

[Tim Gardner (timg-tpi)]

Hmm, this is looking like it might be a binutils issue. Building Ubuntu-3.5.0-14.16 in a Precise chroot produces:

readelf -s ./debian/build/build-generic/arch/x86/boot/setup.elf|grep video_card
   160: 000042ec 0 NOTYPE GLOBAL DEFAULT 12 video_cards
   173: 00004340 0 NOTYPE GLOBAL DEFAULT 12 video_cards_end

Quantal produces:

readelf -s ./debian/build/build-generic/arch/x86/boot/setup.elf|grep video_card
   138: 00003658 0 NOTYPE GLOBAL DEFAULT 12 video_cards
   151: 00003658 0 NOTYPE GLOBAL DEFAULT 12 video_cards_end

[Tim Gardner (timg-tpi)]

Installing binutils_2.22-6ubuntu1_amd64.deb from Precise into the Quantal chroot didn't make any difference. Next I'll try compilers:

rtg@salmon:~/ubuntu/ubuntu-quantal$ cat ./debian.master/abi/3.5.0-14.16/amd64/generic.compiler
GCC: (Ubuntu/Linaro 4.7.1-8ubuntu1) 4.7.1 20120908 (prerelease)

rtg@salmon:~/ubuntu/ubuntu-quantal$ cat ./debian.master/abi/3.5.0-13.13/amd64/generic.compiler
GCC: (Ubuntu/Linaro 4.7.1-7ubuntu1) 4.7.1 20120814 (prerelease)

[Tim Gardner (timg-tpi)]

Doko suggested trying https://launchpad.net/ubuntu/quantal/+source/binutils/2.22.90.20120816-2ubuntu1 with the following positive result:

readelf -s ./debian/build/build-generic/arch/x86/boot/setup.elf|grep video_card
   147: 00002f3b 0 NOTYPE GLOBAL DEFAULT ABS video_cards
   148: 00003650 0 NOTYPE GLOBAL DEFAULT ABS video_cards_end

[Tim Gardner (timg-tpi)]

Indeed, kernels built with binutils 2.22.90.20120816-2ubuntu1 actually appear to work.

[Matthias Klose (doko)]

binutils 2.22.90.20120913-1ubuntu1 is now in the archive, reverting an upstream change. unsure if this a binutils issue or an issure exposed by bintuils. please see http://sourceware.org/ml/binutils/2012-09/msg00159.html for the followup

[Matthias Klose (doko)]

I tried to reproduce this with binutils 2.22.90.20120816-2ubuntu1 on current quantal amd64, however I always see the same offset. with 3.5.0-14.17:

$ KBUILD_VERBOSE=1 DEB_BUILD_OPTIONS="verbose parallel=4" fakeroot debian/rules binary-generic
[...]
$ readelf -s debian/build/build-generic/arch/x86/boot/setup.elf|grep video_cards
   138: 0000365c 0 NOTYPE GLOBAL DEFAULT 12 video_cards
   151: 0000365c 0 NOTYPE GLOBAL DEFAULT 12 video_cards_end

[Jeremy Bicha (jbicha)]

I tested with today's daily build which uses Linux 3.5.0-14.17 and I still get the "undefined video mode" error.

[Matthias Klose (doko)]

Jakub Jelinek commented on freenode/#gcc:

from quick look at the kernel, I'd say the kernel is buggy
I'd say used attribute is a must in the #define __videocard struct card_info __attribute__((section(".videocards")))
because when it declares static __videocard var = { ... }; then the compiler has all rights to just nuke those as unused
though I can't reproduce it with:

struct S { int i, j; };
static struct S __attribute__((section ("foobar"))) s = { 1, 2 };
void foo (void)
{
  s.i = 2;
  s.j = 3;
}

actually, the kernel code is even more insane

struct S { int i, j; void (*p) (void); };
static struct S __attribute__((section ("foobar"))) s;
static void foo (void)
{
  s.i = 2;
  s.j = 3;
}
static struct S __attribute__((section ("foobar"))) s = { 1, 2, foo };

if this ever worked, it was purely by accident
note that the struct var is static, not marked as used, only used (visibly to the compiler) in a static function that is only referenced from the static struct variable's initializer
but please tell me which gcc would not optimize that away, even 4.1 at -O2 does

seems this got broken 4 years ago though
commit a1a00b58855ccdbedf556b4f5638d5208b454472
Author: Hannes Eder <email address hidden>
Date: Sun Nov 23 19:37:09 2008 +0100
a preprocessed testcase still would be nice to see if for some reason it happens not to be optimized out by month old 4.7 and is right now
but the reduced testcase from what I saw in the kernel code really optimizes to nothing even in 4.1
even 3.4 does, 3.3/3.2 doesn't at -O2

[Matthias Klose (doko)]

--- arch/x86/boot/video.h.orig 2012-09-14 14:06:49.005979707 +0000
+++ arch/x86/boot/video.h 2012-09-14 13:43:47.116013331 +0000
@@ -80,7 +80,7 @@
        u16 xmode_n; /* Size of unprobed mode range */
 };

-#define __videocard struct card_info __attribute__((section(".videocards")))
+#define __videocard struct card_info __attribute__((used, section(".videocards")))
 extern struct card_info video_cards[], video_cards_end[];

 int mode_defined(u16 mode); /* video.c */

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.5.0-14.18

---------------
linux (3.5.0-14.18) quantal; urgency=low

  [ Tim Gardner ]

  * SAUCE: Add 'used' to the video_cards structure attributes
    - LP: #1049650
 -- Tim Gardner <email address hidden> Fri, 14 Sep 2012 10:58:40 -0400

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Colin Watson (cjwatson)]

Jeremy's right that this wouldn't affect installed systems with grub2, in general. grub2 uses the kernel's 32-bit entry point, which bypasses the affected code.

[Matthias Klose (doko)]

the pointer to binutils unfortunately wasn't the right one. I always was able to reproduce the issue with any binutils version.
there were section sizes for the .videocards section. 0 (the buggy one), 84 (the correct one), and an even bigger one from comment #8. It was agreed not investigate the even bigger one, as it was only seen by one developer and couldn't be reproduced.

The root cause was a bug in the gcc-4.7 4.7.1-8ubuntu1 package, that the unused video_cards static variables were optimized away, even while the files were built using -fno-toplevel-reorder (which unfortunately was hidden, because quiet build logs were turned on by default). It is likely that Debian #687448 is the same issue. -7 and -7ubuntu1 didn't show the issue, while -8 and -8ubuntu1 did show it, the only change between these two versions being branch updates, however it couldn't be reproduced with an upstream build. The issue couldn't be seen with an update to the recent 4.7 branch (4.7.2 release candidate), which did land as -9ubuntu1 in the archive.

The suggested kernel fix to explicitly mark the video_cards variables as used isn't wrong, but maybe not needed. Please consider rebuilding the linux package using -9ubuntu1/2 when this is available in the archive for all architectures (even armel), so that files which may be affected from the issue are correctly rebuilt.

[Greg Faith (gregfaith)]

Got the same bug on alternate Lubuntu amd64 & i386 (di) installs. Colors were bright and strange lime green & blue but the installs proceeded as usual and completed. Reboots were okay and as expected colors both isos 20120918..

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.