[hardy] kernel 2.6.24-21-generic and gcc version mismatch

[jhansonxi]

Binary package hint: linux-image-2.6.24-21-generic

I'm not sure how it happened but when I updated my Ubuntu 8.04 (Hardy Heron) x86_64 system about a week ago I ended up with:
cat /proc/version
Linux version 2.6.24-21-generic (buildd@crested) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Tue Oct 21 23:09:30 UTC 2008

gcc --version
gcc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu3)

From Synaptic it looks like they both came through hardy-updates. I discovered the mismatch when I attempted to update VMware Player (vmware-config.pl). Since it's installed from source it needs to recompile its module after every kernel update. It pointed out the mismatch and warned against installing. I don't remember how I updated. It either was apt-get or aptitude.

[jhansonxi (jhansonxi)]

I just updated a different system with aptitude that was using the main Ubuntu repo (us.archive.ubuntu.com) and got the same results:

Original versions:
cat /proc/version
Linux version 2.6.24-21-generic (buildd@palmer) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Mon Aug 25 17:32:09 UTC 2008

gcc --version
gcc (GCC) 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

After update:
cat /proc/version
Linux version 2.6.24-21-generic (buildd@palmer) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Tue Oct 21 23:43:45 UTC 2008

gcc --version
gcc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu3)

cat /var/log/aptitude # Pardon the cat abuse
Aptitude 0.4.9: log report
Sun, Nov 2 2008 12:30:13 -0500

IMPORTANT: this log only lists intended actions; actions which fail due to
dpkg problems may not be completed.

Will install 38 packages, and remove 0 packages.
717kB of disk space will be used
===============================================================================
[INSTALL, DEPENDENCIES] libisc35
[UPGRADE] acroread 8.1.2.su1-0.0medibuntu0.8.04.1 -> 8.1.2.su1-0.0medibuntu0.8.04.2
[UPGRADE] acroread-escript 8.1.2.su1-0.0medibuntu0.8.04.1 -> 8.1.2.su1-0.0medibuntu0.8.04.2
[UPGRADE] acroread-plugins 8.1.2.su1-0.0medibuntu0.8.04.1 -> 8.1.2.su1-0.0medibuntu0.8.04.2
[UPGRADE] apt 0.7.9ubuntu17 -> 0.7.9ubuntu17.1
[UPGRADE] apt-utils 0.7.9ubuntu17 -> 0.7.9ubuntu17.1
[UPGRADE] bind9-host 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] cpp-4.2 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] dnsutils 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] g++-4.2 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] gcc-4.2 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] gcc-4.2-base 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] gstreamer0.10-plugins-bad 0.10.6-5 -> 0.10.6-5ubuntu0.1
[UPGRADE] gtk2-engines-pixbuf 2.12.9-3ubuntu4 -> 2.12.9-3ubuntu5
[UPGRADE] libbind9-30 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] libdns35 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] libffi4 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] libgcc1 1:4.2.3-2ubuntu7 -> 1:4.2.4-1ubuntu3
[UPGRADE] libglib2.0-0 2.16.4-0ubuntu3 -> 2.16.6-0ubuntu1
[UPGRADE] libgomp1 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] libgtk2.0-0 2.12.9-3ubuntu4 -> 2.12.9-3ubuntu5
[UPGRADE] libgtk2.0-bin 2.12.9-3ubuntu4 -> 2.12.9-3ubuntu5
[UPGRADE] libgtk2.0-common 2.12.9-3ubuntu4 -> 2.12.9-3ubuntu5
[UPGRADE] libisccc30 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] libisccfg30 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] liblwres30 1:9.4.2-10ubuntu0.1 -> 1:9.4.2.dfsg.P2-2
[UPGRADE] libstdc++6 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] libstdc++6-4.2-dev 4.2.3-2ubuntu7 -> 4.2.4-1ubuntu3
[UPGRADE] linux-headers-2.6.24-21 2.6.24-21.42 -> 2.6.24-21.43
[UPGRADE] linux-headers-2.6.24-21-generic 2.6.24-21.42 -> 2.6.24-21.43
[UPGRADE] linux-image-2.6.24-21-generic 2.6.24-21.42 -> 2.6.24-21.43
[UPGRADE] linux-libc-dev 2.6.24-21.42 -> 2.6.24-21.43
[UPGRADE] pciutils 1:2.2.4-1.1ubuntu5 -> 1:2.2.4-1.1ubuntu6
[UPGRADE] thunderbird-locale-en-gb 1:2.0.0.0+1-0ubuntu1 -> 1:2.0.0.14+1-0ubuntu1~8.04.1
[UPGRADE] tzdata 2008h-0ubuntu0.8.04 -> 2008h-0ubuntu0.8.04.1
[UPGRADE] wine 1.1.6~winehq0~ubuntu~8.04-0ubuntu1 -> 1.1.7~winehq0~ubuntu~8.04-0ubuntu1
[UPGRADE] x...

[Launchpad Janitor (janitor)]

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

[Ante Karamatić (ivoks)]

I can confirm this bug. 64-bit buildd is ouy of sync.

Broken machine (64bit):

# cat /proc/version*
Linux version 2.6.24-23-server (buildd@crested) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Mon Jan 26 01:36:05 UTC 2009
Ubuntu 2.6.24-23.48-server

# gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu3)

On a working machine (32bit):

# cat /proc/version*
Linux version 2.6.24-23-server (buildd@palmer) (gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu3)) #1 SMP Thu Nov 27 19:19:15 UTC 2008
Ubuntu 2.6.24-23.46-server

# gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu3)

[Ante Karamatić (ivoks)]

This is why one can't use modems:

# ./setup
checking for running kernel version...2.6.24
checking for ptserial...ptserial-2.6.c
checking for gcc...4.2.4
checking for kernel gcc version...4.2.3
** error
installed gcc version 4.2.4 does not match kernel gcc version 4.2.3
installing the kernel driver module is not possible
installing a different version of gcc, or your kernel, is necessary
** compilation error

3th party kernel modules can't be built.

[Andy Whitcroft (apw)]

I am assuming you are getting a version missmatch when compiling external modules here. Could you paste in one of those messages, and also attach the /etc/apt/sources.list from both machines please.

[Ante Karamatić (ivoks)]

sources.list for both machine is exactly the same:

Broken:
# egrep -v '^(#|$)' /etc/apt/source.list
deb http://hr.archive.ubuntu.com/ubuntu/ hardy main restricted universe multiverse
deb-src http://hr.archive.ubuntu.com/ubuntu/ hardy main restricted universe multiverse
deb http://hr.archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe multiverse
deb-src http://hr.archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu hardy-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted universe multiverse

Working:
# egrep -v '^(#|$)' /etc/apt/source.list
deb http://hr.archive.ubuntu.com/ubuntu/ hardy main restricted universe multiverse
deb-src http://hr.archive.ubuntu.com/ubuntu/ hardy main restricted universe multiverse
deb http://hr.archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe multiverse
deb-src http://hr.archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu hardy-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted universe multiverse

[Ante Karamatić (ivoks)]

Note that I didn't reboot 32bit machine yet, that's why /proc/version_signature is 46. It's a production machine and i can't reboot it now.

[Andy Whitcroft (apw)]

This appears to be a compiler missmatch for users of -updates triggered by the fact that the kernel in question was a security upload to -security, which necessarily was built using only packages from -release + -security.

Basically we either need to update the gcc in -security or have separate kernel builds in -security and -updates. Neither is trivial. Wil start discussions on how to resolve this on the kernel-team mailing list.

[KarlGoetz (kgoetz)]

I've looked at [1] and [2] and dont find this discussion. Could you point me to it?

[1] https://lists.ubuntu.com/archives/kernel-team/2009-February/thread.html
[2] https://lists.ubuntu.com/archives/kernel-team/2009-March/thread.html

[Karl M. Davis (karlmdavis)]

Just thought I'd throw in a "me too" on this one.

For those Googling about this problem, here's the error I received from vmware-config-tools.pl that led me to this bug report:
<<
# vmware-config-tools.pl
...
None of the pre-built vmhgfs modules for VMware Tools is suitable for your
running kernel. Do you want this program to try to build the vmhgfs module for
your system (you need to have a C compiler installed on your system)? [yes]

Using compiler "/usr/bin/gcc". Use environment variable CC to override.

Your kernel was built with "gcc" version "4.2.3", while you are trying to use
"/usr/bin/gcc" version "4.2.4". This configuration is not recommended and
VMware Tools may crash if you'll continue. Please try to use exactly same
compiler as one used for building your kernel. Do you want to go with compiler
"/usr/bin/gcc" version "4.2.4" anyway? [no]
...
>>

[Chris Sherlock (ta-bu-shi-da-yu)]

Looks like it's happened again.

I just filed the following bug report, this time it's occured on Ibex:

https://bugs.launchpad.net/ubuntu/+source/gcc-defaults/+bug/343614

[Andy Whitcroft (apw)]

As Chris indicates we appear now to have a missmatch on Intrepid. Note that in the -security pocket the gcc version is lower than that in -updates but we have the same kernel in each of those two pockets.

Intrepid GCC
    -release 4.3.2-1ubuntu11
    -updates 4.3.2-1ubuntu12

Intrepid Kernel
    -release 2.6.27-7.14
    -security 2.6.27-11.27
    -updates 2.6.27-11.27
    -proposed 2.6.27-14.30

For clarity here is the Hardy case:

Hardy GCC
    -release 4.2.3-2ubuntu7
    -updates 4.2.4-1ubuntu3
    -proposed 4.2.4-1ubuntu4

Hardy Kernel
    -release 2.6.24-16.30
    -security 2.6.24-23.48
    -updates 2.6.24-23.48
    -proposed 2.6.24-24.50

Note that we have an additional problem that the kernel in -proposed is likely compiled with the compiler from -proposed and therefore they both need to migrate to -updates together.

[Colin Watson (cjwatson)]

We should consider Intrepid separately since it didn't bump the upstream compiler version, which suggests to me as though it might be something different. Andy, I thought you said earlier that the upstream compiler version got encoded in the kernel's ABI, not the whole package version?

[Colin Watson (cjwatson)]

I've copied gcc-4.2 from hardy-updates to hardy-security now. Kees is going to issue a USN.

[KarlGoetz (kgoetz)]

I've not seen a USN for it yet. Any timeframe for it arriving?
kk

[Chris Sherlock (ta-bu-shi-da-yu)]

This same issue is now affecting Jaunty, which means that it's so far affected hardy, intrepid and jaunty.

I was wondering how things were going with resolving this issue? It's making it impossible to compile Cisco's VPN kernel module.

[Ante Karamatić (ivoks)]

@Chris
How does it affect jaunty? Looking at the gcc versions (4.3.3-5ubuntu4) of gcc-4.3 package and in /proc/version, they are the same. Are you using -proposed?

[Chris Sherlock (ta-bu-shi-da-yu)]

I must make my apologies... I misread /proc/version.

My apologies for the bug spam... how embarassing!

[Andy Whitcroft (apw)]

This should now be sorted going forward as we should no longer be allowing out of sync compilers in the security pocket.