Installer needs way to install PAE kernel on i386 9.10 DVD

Reported by Jerone Young on 2009-08-13
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OEM Priority Project
High
Unassigned
base-installer (Ubuntu)
High
Colin Watson
Karmic
High
Colin Watson
ubiquity (Ubuntu)
High
Evan Dandrea
Karmic
High
Evan Dandrea

Bug Description

The 9.10 DVD has the pae kernel packages on the media. Though the installer has no way to install it.

The proposed way is to have it so that installer can read that the machine has greater then 3Gigs of memory and CPU supports the PAE bit to install the PAE kernel automatically or have an option so it can be installed.

Jerone Young (jerone) on 2009-08-13
visibility: private → public
Changed in oem-priority:
assignee: nobody → Canonical Ubuntu QA Team (canonical-qa)
Colin Watson (cjwatson) on 2009-08-24
Changed in base-installer (Ubuntu):
assignee: nobody → Colin Watson (cjwatson)
importance: Undecided → High
status: New → Triaged
Colin Watson (cjwatson) wrote :

I'll make it do the right thing by default for machines with >3GB of RAM. If you want more control on other machines with less memory than that, you can use expert mode, or preseed base-installer/kernel/override-image.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package base-installer - 1.101ubuntu4

---------------
base-installer (1.101ubuntu4) karmic; urgency=low

  * Prefer PAE kernels on machines with >3GB of RAM (LP: #413135).

 -- Colin Watson <email address hidden> Mon, 24 Aug 2009 13:26:20 +0100

Changed in base-installer (Ubuntu):
status: Triaged → Fix Released
Changed in oem-priority:
assignee: Canonical Ubuntu QA Team (canonical-qa) → nobody
Jerone Young (jerone) on 2009-08-30
Changed in base-installer (Ubuntu):
status: Fix Released → New
Jerone Young (jerone) wrote :

This problem is not fixed. Trying DVD with d-i installer from Aug 29 on a machine with 4gigs of memory, it did not install the PAE kernel.

If you are using /proc/meminfo , remember that only displays the amount of memory the kernel can see. So any tool dependent on that data will fail.

You can try "dmidecode" and under the ""Memory Array Mapped Address" section in value "Range Size" . You can easily find the amount of memory on the machine. Will require some greping but is possible.

Changed in oem-priority:
assignee: nobody → Canonical Ubuntu QA Team (canonical-qa)
Changed in base-installer (Ubuntu):
status: New → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package base-installer - 1.101ubuntu5

---------------
base-installer (1.101ubuntu5) karmic; urgency=low

  * Use DMI information if possible to determine memory size (LP: #413135).

 -- Colin Watson <email address hidden> Wed, 23 Sep 2009 16:24:16 +0100

Changed in base-installer (Ubuntu):
status: Confirmed → Fix Released
Jerone Young (jerone) on 2009-09-28
Changed in oem-priority:
importance: Undecided → High
Jerone Young (jerone) wrote :

Great work Collin that works!

Though I had some misunderstanding when filing this bug initially. This logic is also need in Ubiquity. As the factory install process for standard Ubuntu inside Dell uses Ubiquity and not the debian installer (my bad, point of confusion with different Ubuntu type releases).

Instead of opening a new bug I'll put as also effect Ubiquity.

Jerone Young (jerone) on 2009-09-29
Changed in oem-priority:
status: New → In Progress

On Tue, Sep 29, 2009 at 05:26:33PM -0000, Jerone Young wrote:
> Though I had some misunderstanding when filing this bug initially. This
> logic is also need in Ubiquity. As the factory install process for
> standard Ubuntu inside Dell uses Ubiquity and not the debian installer
> (my bad, point of confusion with different Ubuntu type releases).

Ubiquity already uses the same code to do kernel selection, and should
include this change as of version 1.99.22. Have you verified this not to
work in a Ubiquity-based installation? If so, I'd appreciate
installation logs.

Jerone Young (jerone) wrote :

Yes Ubiquity is not installing the PAE kernel. Though debian installer is. I've attached logs.

Jerone Young (jerone) wrote :

Just caught another issue. While the pae kernel is being installed. The linux-headers-*-generic-pae are NOT being installed. The package is on the DVD.

The linux-headers-*-generic are being installed should not be.

The issue here is dkms packages will fail to build without the proper header package installed. I'll file another bug if needed.

Changed in base-installer (Ubuntu):
status: Fix Released → Confirmed
Changed in ubiquity (Ubuntu):
assignee: nobody → Colin Watson (cjwatson)
status: New → Confirmed
importance: Undecided → High
Colin Watson (cjwatson) wrote :

Right now, the only way that linux-headers-* packages get installed is
because they're recommends of the desktop metapackage. I'm a bit
cautious about reorganising this for karmic at this point; it is
delicate.

Jerone, is there a particular reason you're insisting (by reopening the
bug task) that the base-installer installer component should be
responsible for doing this? It would certainly be one possible way to
implement it, but it's not the only one.

Jerone Young (jerone) wrote :

@Colin
   We do need a solution then for this issue with the PAE kernel. DKMS packages are something that we have to often use. This can be solved by hacking the linux-headers-pae into a preseed file. Though this is something that will cause many problems down the road as some will be unaware why DKMS packages are failing when installing to a system that has the PAE kernel.

  Yes I reopen the bug task since the problem is not fully fixed. I was unaware that desktop metapackage was the one installing the linux-headers. I can open a new bug for the pae headers.

Colin Watson (cjwatson) wrote :

Yes, but you reopened the bug task on a very specific package - in
complex cases such as this with multiple tasks it's better to just leave
a comment and let a developer work out where it needs to go, I think.

I don't think you need to open a new bug at this time.

Jerone Young (jerone) wrote :

@Colin
      Will do that in the future. Got some feedback though on the headers. A bit of push back asking "Why are the headers not recommended by the kernel package?" also that "it's an easy and needed fix".

      Can we try to get this in before the next RC release so can be tested to be ok.

Jerone Young (jerone) wrote :

Any progress on this bug?

Colin Watson (cjwatson) wrote :

I have accepted this bug and intend to work on it before Ubuntu final. Beyond that I'm afraid I'm not able to give an exact timescale right now.

Colin Watson (cjwatson) wrote :

(obviously that should have read 9.10 final)

Jerone Young (jerone) wrote :

This bug needs to be resolved in Final. Colin do you need to work with the kernel team to have the packages correctly pull in the correct headers.

This issue effects more the oem here. But any user who installs from the i386 DVD & more the 3GB RAM (which means PAE kernel gets installed) and wants to use proprietary drivers. They will have the wrong headers installed and will fail.

This a bug that cannont be fixed post final.

Colin Watson (cjwatson) wrote :

I do not need to work with the kernel team. This does not need a kernel
change. If I change my mind, I know where to find the kernel team at
short notice.

Why, when I say "I intend to work on this for 9.10 final", do you need
to repeat twice that this needs to be resolved in final? Please leave me
to get on with my work - I've targeted this bug in such a way that it
will not be forgotten.

Colin Watson (cjwatson) on 2009-10-13
Changed in base-installer (Ubuntu Karmic):
status: Confirmed → In Progress
Changed in base-installer (Ubuntu Karmic):
milestone: none → ubuntu-9.10
Colin Watson (cjwatson) on 2009-10-13
Changed in base-installer (Ubuntu Karmic):
status: In Progress → Fix Committed
Changed in ubiquity (Ubuntu Karmic):
milestone: none → ubuntu-9.10
status: Confirmed → In Progress
Colin Watson (cjwatson) wrote :

For the record, doing this by way of a Recommends in the kernel package
would have had zero effect on d-i's behaviour. The installer
intentionally ignores linux-image-* Recommends, because they include
boot loaders that it needs to have more explicit control over (although
these Recommends are correct for a normal system).

ubiquity needs an independent fix, although along similar lines to that
in base-installer.

Colin Watson (cjwatson) on 2009-10-13
Changed in ubiquity (Ubuntu Karmic):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package base-installer - 1.102ubuntu2

---------------
base-installer (1.102ubuntu2) karmic; urgency=low

  * Install kernel headers to match the kernel (LP: #413135). This may be
    overridden by setting base-installer/kernel/headers to false.

 -- Colin Watson <email address hidden> Tue, 13 Oct 2009 20:47:08 +0100

Changed in base-installer (Ubuntu Karmic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 2.0.0

---------------
ubiquity (2.0.0) karmic; urgency=low

  [ Colin Watson ]
  * Bump to a stable version number series for Ubuntu 9.10.

    The main reason for the version bump was because we incorporated
    oem-config and needed to jump past its version numbers, but really
    Ubiquity 2.0.0 is quite different from Ubiquity 1.0.0 despite most of
    the changes being gradual and incremental. 3.5 years and nearly 5000
    lines of changelog entries later, we have several active developers,
    lots more translations, apport integration, a new partitioner,
    migration-assistant, automation, Mythbuntu and noninteractive frontends,
    Wubi, reinstallation without reformatting, a new timezone map,
    accessibility, many UI improvements, a slideshow, plugin support, and
    much more. Here's to version 3!

  * Install kernel headers to match the kernel (LP: #413135).
  * Recommend dmraid to ensure consistent behaviour across Ubuntu flavours
    (it was already present on the Ubuntu desktop CD, but e.g. not on
    Kubuntu).
  * If dmraid devices are active, then create
    /var/lib/disk-detect/activate_dmraid so that partman will exclude slave
    devices, and ensure that dmraid is installed in the target (LP: #90235).
  * Use check-language-support if available to select language support
    packages (LP: #434173).
  * Update translations from Launchpad.
  * Automatic update of included source packages: base-installer
    1.102ubuntu2, hw-detect 1.72ubuntu5.

  [ Jonathan Riddell ]
  * ubi-intro.py: detect kubuntu-netbook-intro.txt if it exists and use
    that as warning for Kubuntu Netbook Edition

 -- Colin Watson <email address hidden> Wed, 14 Oct 2009 23:57:40 +0100

Changed in ubiquity (Ubuntu Karmic):
status: Fix Committed → Fix Released
Changed in oem-priority:
assignee: Canonical Ubuntu QA Team (canonical-qa) → nobody
Jerone Young (jerone) wrote :

Trying DVD spin from October 17 .

Ubiquity is NOT fixed.
1) Not installing PAE kernel
2) Not installing PAE kernel headers

Thoug Debian Installer is fixed and doing the proper things.

Changed in ubiquity (Ubuntu Karmic):
status: Fix Released → Confirmed
Colin Watson (cjwatson) wrote :

On Mon, Oct 19, 2009 at 07:16:09AM -0000, Jerone Young wrote:
> Ubiquity is NOT fixed.
> 1) Not installing PAE kernel
> 2) Not installing PAE kernel headers

Please (urgently) attach full logs: I need /var/log/installer/syslog.

Colin Watson (cjwatson) wrote :

Never mind; I think I managed to figure this out. Thanks for the heads-up.

Changed in ubiquity (Ubuntu Karmic):
status: Confirmed → Fix Committed
Jerone Young (jerone) wrote :
Jerone Young (jerone) wrote :
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 2.0.1

---------------
ubiquity (2.0.1) karmic; urgency=low

  [ Evan Dandrea ]
  * Check to see whether X crashed or ubiquity crashed before attempting
    to bail into the noninteractive frontend from only-ubiquity mode
    (LP: #444901).
  * Wrap the format warning label while working around GTK+ label layout
    problems (LP: #364617).

  [ Colin Watson ]
  * Rewrite X_LOADTEMPLATEFILE commands in case they refer to template files
    in the /target chroot (LP: #452118).
  * Install -generic-pae kernels if necessary; this requires some custom
    hacks since -generic-pae isn't in the live filesystem (LP: #413135).
  * Don't do kernel installation/removal in oem-config.
  * Skip partition_too_small check during Wubi installs, as Wubi does some
    of its own checks and the delay imposed by this check looks particularly
    weird in Wubi. This may or may not be the cause of apparent hangs
    towards the end of partitioning, but I suspect that it will at least get
    rid of some conflated reports and make testing quicker.
  * Update translations from Launchpad.
  * Automatic update of included source packages: clock-setup 0.98ubuntu3,
    flash-kernel 2.13ubuntu13, grub-installer 1.43ubuntu6, partman-auto
    89ubuntu2, partman-target 64ubuntu4.

  [ Mario Limonciello ]
  * Mythbuntu: Don't allow "removing" LIRC as it's not supported with install.

  [ Roman Shtylman ]
  * Fixed India timezone map (LP: #453009)

  [ Michael Casadevall ]
  * Added support for partman-uboot in ubiquity (LP: #455713)

 -- Colin Watson <email address hidden> Tue, 20 Oct 2009 01:20:06 +0100

Changed in ubiquity (Ubuntu Karmic):
status: Fix Committed → Fix Released
Brent Fox (brent-s-fox) on 2009-10-21
Changed in dell:
assignee: nobody → Jerone Young (jerone)
importance: Undecided → High
status: New → In Progress
Jerone Young (jerone) wrote :

This issue is still not resolved. Running 10/20/09 DVD with Ubiquity 2.0.2 . It is still NOT
- installing PAE kernel
- installing PAE kernel headers

on a machine with 3gigs or more memory.

Need for developer to test fixes before shipping them out, as delta between testing is taking to long.

Changed in ubiquity (Ubuntu Karmic):
status: Fix Released → Confirmed
Jerone Young (jerone) wrote :

This issue is still not resolved. Running 10/20/09 DVD with Ubiquity 2.0.2 . It is still NOT
- installing PAE kernel
- installing PAE kernel headers

on a machine with 3gigs or more memory.

Need for developer to test fixes before shipping them out, as delta between testing is taking to long.

Colin Watson (cjwatson) wrote :

Sorry, I have no way to test fixes; I simply don't have the space or
bandwidth. This approach is the best I can manage and I would appreciate
your forbearance.

Colin Watson (cjwatson) wrote :

For the record, I actually did test this, just in a slightly bodged test
rig rather than on real hardware. Sometimes installer testing doesn't
quite carry over to the real world easily.

What does 'sudo /usr/lib/base-installer/dmi-available-memory' say on the
machine you tested on, when running the live DVD session?

Jerone Young (jerone) wrote :

I'll have some info later. What I can say now is the same machine works correctly with the Debian installer. It will install the PAE kernel & headers. Though Ubiquity will not. So the script (assuming D-I uses it) may not be the issue.

Colin Watson (cjwatson) wrote :

Both d-i and ubiquity use dmi-available-memory, but they have
substantially different implementations of most of the other stuff here
(necessarily). You're right that this does tend to suggest that
dmi-available-memory is not at fault, though.

Jerone Young (jerone) wrote :

The dmi-available-memory script is working fine. I've test it on Dell systems.. as well as the test system I am using now (a Lenovo Thinkpad T61). All report the available memory. The Dell systems only had 2 - 3 gigs of memory. So I have to use my Thinkpad T61 for testing.

For my Thinkpad T61 with 4G RAM produces output:
4194304

While the output of a Dell XPS 13 with 2G RAM produces output:
2097152

Given the d-i installer has no issue think the issue may be further in the logic of ubiquity.

Jono Bacon (jonobacon) wrote :

Please note, we have put together a test case at http://testcases.qa.ubuntu.com/Hardware/I386%2032-bit%20PAE and I have asked the community to get involved. I hope this generates some more data.

Changed in ubiquity (Ubuntu Karmic):
assignee: Colin Watson (cjwatson) → Evan Dandrea (evand)
Tobin Davis (gruemaster) wrote :

I had a friend at Intel test the cd image yesterday from http://cdimage.ubuntu.com/daily-live/20091020.3/karmic-desktop-i386.iso and he reported that on a 8x Prestonia (Xeon) system with 6G memory, it correctly reported the right amount in K. Not sure if this helps, as it was a server type system instead of a desktop and not the dvd image.

Tobin Davis (gruemaster) wrote :

I had a friend at Intel test the http://cdimage.ubuntu.com/daily-live/20091020.3/karmic-desktop-i386.iso image yesterday. He reports that on a 8x Prestonia (Xeon) system with 6G memory, the correct amount of memory is being reported by 'sudo /usr/lib/base-installer/dmi-available-memory'.

Jerone Young (jerone) wrote :

@Tobin
     That reaffirms dmi-available-memory is doing the correct thing. For a wide variety of systems. So something is up in Ubiquity then and not external to it. So where we stand now with both installers using dmi-available-memory is:

Ubiquity -> Not working

Debian Installer -> working fine

Kees Cook (kees) wrote :

(As a tangent, it might be easier to just install the PAE kernels if the "pae" flag exists in /proc/cpuinfo. Without that flag, the amount of memory isn't interesting. With that flag, they should get a kernel that supports future memory upgrades.)

Matt Zimmerman (mdz) on 2009-10-22
Changed in ubiquity (Ubuntu Karmic):
status: Confirmed → In Progress
Evan Dandrea (ev) wrote :

This appears to be a matter of missing code to determine the amount of available memory. I simply took the code to do this from base-installer/library.sh and added it to ubiquity's check-kernels script (see attached patch). I've uploaded a version of ubiquity with this fix to my ppa:

deb http://ppa.launchpad.net/evand/ppa/ubuntu karmic main

Please do the following to test this:
1) Boot the "Try or Install Ubuntu" option off the i386 DVD.
2) Add the above line to your sources.list.
3) Run `sudo apt-get update; sudo apt-get install ubiquity-frontend-gtk=2.0.3~evand`
4) Launch ubiquity as usual, you should get a pae kernel on reboot.

Evan Dandrea (ev) wrote :

Do note that you'll need more than 3GB of available memory to get the PAE kernel.

Tim Gardner (timg-tpi) wrote :

Evan - I really like Kee's suggestion of detecting PAE capability in the CPU.

Evan Dandrea (ev) wrote :

Kees, Tim, the code checks for both the PAE CPU flag and for more than 3145728K of available memory.

Evan Dandrea (ev) wrote :

Jerone has reported a successful install. He noted that the generic kernel was also present, which I'm looking into.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 2.0.3

---------------
ubiquity (2.0.3) karmic; urgency=low

  [ Colin Watson ]
  * Use new check-language-support --show-installed option added in
    language-selector 0.4.16, so that we can arrange to keep language
    support packages installed that are already present in the live
    filesystem.

  [ Mario Limonciello ]
  * Pass the debug parameters to ubiquity even if running in noninteractive
    mode.
  * Don't provide the old init scripts, even under a temporary name anymore.
    - It appears that there is a race condition that can exist between when
      the upstart'ified version and the initscript version start that may cause
      only-ubiquity and automatic-ubiquity to (poorly) fail. (LP: #457858)
    - All login managers in use (gdm and kdm) have converted to upstart now.

  [ Evan Dandrea ]
  * Calculate the amount of available memory in check-kernels (LP: #413135).
  * Automatic update of included source packages: grub-installer
    1.43ubuntu7.

 -- Evan Dandrea <email address hidden> Fri, 23 Oct 2009 08:11:00 +0100

Changed in ubiquity (Ubuntu Karmic):
status: In Progress → Fix Released
Evan Dandrea (ev) wrote :

Reopening this to track the issue of the generic kernel still being installed.

Changed in ubiquity (Ubuntu Karmic):
status: Fix Released → Triaged
Colin Watson (cjwatson) wrote :

On Thu, Oct 22, 2009 at 04:49:09PM -0000, Kees Cook wrote:
> (As a tangent, it might be easier to just install the PAE kernels if the
> "pae" flag exists in /proc/cpuinfo. Without that flag, the amount of
> memory isn't interesting. With that flag, they should get a kernel that
> supports future memory upgrades.)

I've stated my position on this before in an e-mail thread, although it
doesn't look as though it made it into this bug.

PAE is slow. I realise that it has some security benefits, but it is
SLOW. Linus has some fairly excoriating comments on it here (HIGHMEM64G
== PAE) - I don't fancy the idea of making forks significantly more
expensive, especially considering the focus on boot speed:

  http://article.gmane.org/gmane.linux.kernel/845637

As such, I strongly feel that we have to make its use conditional on
having enough memory to require it. The memory upgrade issue is
unfortunate, but with creativity we could address it in other ways.

Loïc Minier (lool) wrote :

@Evan: do we need to keep the remainder of this bug as a karmic milestoned task or can it be deferred to lucid?

I don't think it's a big deal if big iron servers need to have space to install two kernels instead of one, unless the non-pae kernel might get preferred on boot (is the pae kernel preferred right now?).

Evan Dandrea (ev) wrote :

Loic, there are two items left to reach parity. One is to not install the regular kernel, which, as you suggest, isn't a big deal. The other is to properly identify the kernel version in ubiquity, so that the /vmlinuz and /initrd symlinks get created. I have fixes for both of these that I've successfully tested. I'm just running through one more test install.

Evan Dandrea (ev) wrote :

Marc Tardif reported a successful install with the patch that went into 2.0.3.

Evan Dandrea (ev) wrote :

The following patch fixes the aforementioned remaining issues for me. If we can get another successful test on this, I'd like to upload it for Karmic. The code path this touches only affects machines that meet the requirements for PAE, so I think it's pretty safe.

Evan Dandrea (ev) wrote :

I've uploaded a version of ubiquity to my PPA with this patch:

deb http://ppa.launchpad.net/evand/ppa/ubuntu karmic main

Please do the following to test this:
1) Boot the "Try or Install Ubuntu" option off the i386 DVD.
2) Add the above line to your sources.list.
3) Run `sudo apt-get update; sudo apt-get install ubiquity-frontend-gtk=2.0.4~evand`
4) Launch ubiquity as usual

You should get only a PAE kernel on reboot, and the /vmlinuz and /initrd.gz symlinks should point to the PAE kernel and initramfs.

Evan Dandrea (ev) wrote :

Jerone reported a successful install with 2.0.4~evand

Evan Dandrea (ev) on 2009-10-23
Changed in ubiquity (Ubuntu Karmic):
status: Triaged → Fix Committed
Tony Espy (awe) wrote :

Worked for me too using the PPA.

Patrick Nell (patrick-nell) wrote :

Worked for me too using the PPA.

Evan Dandrea (ev) wrote :

This is just waiting for an archive admin to process ubiquity 2.0.4.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 2.0.4

---------------
ubiquity (2.0.4) karmic; urgency=low

  [ Evan Dandrea ]
  * Do not install the live CD kernel in the target system when using
    PAE (LP: #413135).
  * Properly set the kernel version in the main install process when
    using PAE so that symlinks get created for the kernel and initramfs.
  * Automatic update of included source packages: partman-base
    133ubuntu4.
  * Automatic update of included source packages: grub-installer
    1.43ubuntu8, partman-basicmethods 43ubuntu1.

  [ Colin Watson ]
  * Change apport hook to prefer syslog, partman, and casper.log from
    /var/log/installer/ if they exist there, to support bug-filing after
    installation.
  * Don't set the "incomplete language support" note if only gimp-help-* is
    missing, since it's far too big to fit on CDs (LP: #452516).
  * Furthermore, always consider English as "complete enough". The packages
    that are missing from an installation from the Ubuntu desktop CD are not
    critical for a reasonable user experience.
  * Make use of check-language-support -a if pkgsel/language-packs is ALL,
    since that's orders of magnitude faster (LP: #458333).
  * Keep language-support-$LL installed if it happens to be in the live
    filesystem, since there's no point spending time removing it; but don't
    install it if it isn't in the live filesystem (LP: #458333).

 -- Evan Dandrea <email address hidden> Sat, 24 Oct 2009 14:55:26 +0100

Changed in ubiquity (Ubuntu Karmic):
status: Fix Committed → Fix Released
Jerone Young (jerone) on 2009-11-02
Changed in oem-priority:
status: In Progress → Fix Released
Changed in dell:
status: In Progress → Fix Released
Changed in somerville:
assignee: nobody → Jerone Young (jerone)
importance: Undecided → High
status: New → Fix Released
no longer affects: dell
Timothy R. Chavez (timrchavez) wrote :

The bug task for the somerville project has been removed by an automated script. This bug has been cloned on that project and is available here: https://bugs.launchpad.net/bugs/1305535

no longer affects: somerville
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers