hd-media installer rejects official raring ISO

[Nathan Dorfman]

Following the Ubuntu 13.04 Installation Guide's USB install instructions for amd64 at https://help.ubuntu.com/13.04/installation-guide/amd64/ch04s03.html#usb-copy-flexible doesn't work, it seems. I guess not many people are using the hd-media + ISO install method? The ISO is located and mounted successfully, then rejected due to missing items in the Packages file.

What I did:

1) Downloaded vmlinuz and initrd.gz to a USB drive from
http://archive.ubuntu.com/ubuntu/dists/raring/main/installer-amd64/current/images/hd-media/

2) Set up syslinux to boot those two

3) Downloaded to the same flash drive ubuntu-13.04-desktop-amd64.iso

4) Booted from the flash drive.

After the ISO is located and mounted, the installer errors out and is unable to proceed. The log says:

Sep 2 17:24:55 iso-scan: ISOS_FOUND='[sdf1]
/isos/ubuntu-13.04-desktop-amd64.iso (raring - 13.04)'
Sep 2 17:24:55 iso-scan: Selected ISO:
/isos/ubuntu-13.04-desktop-amd64.iso on /dev/sdf1
Sep 2 17:24:55 kernel: [ 50.033763] ISO 9660 Extensions: Microsoft
Joliet Level 3
Sep 2 17:24:55 kernel: [ 50.036768] ISO 9660 Extensions: RRIP_1991A
Sep 2 17:24:55 iso-scan: Detected ISO with 'raring' (raring) distribution
Sep 2 17:24:55 iso-scan: Detected ISO with distribution 'raring' (raring)
Sep 2 17:24:55 anna-install: Queueing udeb apt-mirror-setup for later
installation
Sep 2 17:24:55 anna-install: Queueing udeb apt-cdrom-setup for later
installation
Sep 2 17:24:55 anna-install: Queueing udeb raring-support for later
installation
Sep 2 17:24:55 main-menu[423]: (process:1616): libkmod:
kmod_lookup_alias_from_builtin_file: could not open builtin file
'/lib/modules/3.8.0-19-generic/modules.builtin.bin'
Sep 2 17:24:55 main-menu[423]: (process:1616): [sdf1]
isos/ubuntu-13.04-desktop-amd64.iso (raring - 13.04)
Sep 2 17:24:55 main-menu[423]: DEBUG: resolver (libc6-udeb): package
doesn't exist (ignored)
Sep 2 17:24:55 main-menu[423]: INFO: Menu item 'load-iso' selected
Sep 2 17:24:55 cdrom-retriever: warning: Unable to find
main/debian-installer/binary-amd64/Packages in
/cdrom/dists/raring/Release.
Sep 2 17:24:55 cdrom-retriever: warning: Unable to find
main/debian-installer/binary-amd64/Packages.gz in
/cdrom/dists/raring/Release.
Sep 2 17:24:55 cdrom-retriever: warning: Unable to find
restricted/debian-installer/binary-amd64/Packages in
/cdrom/dists/raring/Release.
Sep 2 17:24:55 cdrom-retriever: warning: Unable to find
restricted/debian-installer/binary-amd64/Packages.gz in
/cdrom/dists/raring/Release.
Sep 2 17:25:00 anna[3819]: WARNING **: bad d-i Packages file
Sep 2 17:25:00 main-menu[423]: INFO: Menu item 'load-iso' succeeded
but requested to be left unconfigured.

[Nathan Dorfman (ndorf)]

Sorry, I think this is the right package now, yes?

[Quinn Balazs (qbalazs)]

The issue here is not actually with anna, the problem seems from the bit you posted to be cdrom-retriever.

[Nathan Dorfman (ndorf)]

Thanks.

I'm pretty sure I'm using the right files, here the local sha1sums:

3eb9f59aab6e6c8714b98cb8553f9c8ac9a5bcbd boot/ubuntu/raring/amd64/hd-media/initrd.gz
552b3d59f14b0ec9787af84e4f9527bd1966d51a boot/ubuntu/raring/amd64/hd-media/vmlinuz
ffed440f1dc1b43d9c170bd21e5ff669a59447f8 isos/ubuntu-13.04-desktop-amd64.iso

[Quinn Balazs (qbalazs)]

Sha1sums check out. I'll check it out tonight and see if I can manage to duplicate the issue.

[Nathan Dorfman (ndorf)]

I just tried this with saucy beta1 (gnome edition, as the unity one isn't available yet), with the same result:

Sep 5 22:40:11 iso-scan: ISOS_FOUND='[sdf1] /isos/ubuntu-gnome-13.10-beta1-desktop-amd64.iso (saucy - 13.
10)'
Sep 5 22:40:11 iso-scan: Selected ISO: /isos/ubuntu-gnome-13.10-beta1-desktop-amd64.iso on /dev/sdf1
Sep 5 22:40:11 kernel: [ 13.982452] ISO 9660 Extensions: Microsoft Joliet Level 3
Sep 5 22:40:11 kernel: [ 13.984637] ISO 9660 Extensions: RRIP_1991A
Sep 5 22:40:11 iso-scan: Detected ISO with 'saucy' (saucy) distribution
Sep 5 22:40:11 iso-scan: Detected ISO with distribution 'saucy' (saucy)
Sep 5 22:40:11 anna-install: Queueing udeb apt-mirror-setup for later installation
Sep 5 22:40:11 anna-install: Queueing udeb apt-cdrom-setup for later installation
Sep 5 22:40:11 anna-install: Queueing udeb saucy-support for later installation
Sep 5 22:40:11 main-menu[298]: (process:1493): libkmod: kmod_lookup_alias_from_builtin_file: could not op
en builtin file '/lib/modules/3.11.0-4-generic/modules.builtin.bin'
Sep 5 22:40:11 main-menu[298]: (process:1493): [sdf1] isos/ubuntu-gnome-13.10-beta1-desktop-amd64.iso (saucy - 13.10)
Sep 5 22:40:11 main-menu[298]: DEBUG: resolver (libc6-udeb): package doesn't exist (ignored)
Sep 5 22:40:11 main-menu[298]: INFO: Menu item 'load-iso' selected
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find main/debian-installer/binary-amd64/Packages in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find main/debian-installer/binary-amd64/Packages.gz in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find restricted/debian-installer/binary-amd64/Packages in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find restricted/debian-installer/binary-amd64/Packages.gz in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find universe/debian-installer/binary-amd64/Packages in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find universe/debian-installer/binary-amd64/Packages.gz in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find multiverse/debian-installer/binary-amd64/Packages in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 cdrom-retriever: warning: Unable to find multiverse/debian-installer/binary-amd64/Packages.gz in /cdrom/dists/saucy/Release.
Sep 5 22:40:34 anna[3006]: WARNING **: bad d-i Packages file

Curiously, the saucy installer preferred the 13.04 iso over the 13.10 one when it could find both. In order to get it to even try the 13.10 iso, I had to rename the 13.04 one so it wouldn't be found, otherwise:

Sep 5 22:16:57 iso-scan: ISOS_FOUND='[sdf1] /isos/ubuntu-13.04-desktop-amd64.iso (raring - 13.04), [sdf1]
 /isos/ubuntu-gnome-13.10-beta1-desktop-amd64.iso (saucy - 13.10)'
Sep 5 22:16:57 iso-scan: Selected ISO: /isos/ubuntu-13.04-desktop-amd64.iso on /dev/sdf1
Sep 5 22:16:57 kernel: [ 34.274321] ISO 9660 Extensions: Microsoft Joliet Level 3
Sep 5 22:16:57 kernel: [ 34.276189] ISO 9660 Extensions: RRIP_1991A
Sep 5 22:16:57 iso-scan: Detected ISO with 'raring' (raring) distributi...

[Quinn Balazs (qbalazs)]

I have duplicated this using hd-media with both 13.04 and saucy beta1. I believe that this is to be dealt with in debugs, but I will figure that out for sure and then let you know.

[Nathan Dorfman (ndorf)]

Thanks. If there is anything else I can do, let me know.

[Quinn Balazs (qbalazs)]

After asking around, the general consensus was that this belonged on debugs. Since I was able to reproduce it, either I can file it there or I will attach some instructions here for you to do it. Let me know what is most convenient for you.

Quinn Balazs

[Nathan Dorfman (ndorf)]

Well, I just discovered that the debian wheezy iso has those main/debian-installer files, so I bet their installer would work. I'll try it.

This must have something to do with the old alternate install CD, no? I bet they used to have these files as well, and were needed to work with the hd-media installer. I could try with the 12.04 alternate iso.

I guess either the desktop isos would need some more files to work with this method, or Ubuntu would need its own version of the hd-media initrd that could actually use the Ubiquity installer? That'd be pretty awesome.

OTOH it could also just be removed from the install guide. At the end of the day, the netboot image works fine to boot from USB and install saucy. Personally I think it's pretty convenient to have the ISO there as well, but it doesn't look like anyone but me is using this method if it's been broken for this long.

[Nathan Dorfman (ndorf)]

Yep, the 12.04 alternate iso has these files:

dists/precise% ls -lR main/debian-installer
main/debian-installer:
total 2
dr-xr-xr-x 1 root root 2048 Aug 20 18:46 binary-amd64

main/debian-installer/binary-amd64:
total 47
-r--r--r-- 1 root root 47446 Aug 20 18:46 Packages.gz
dists/precise% fgrep -c debian-installer Release
12

[Quinn Balazs (qbalazs)]

Changed to reflect the fact that this is an issue with install documentation.

[Doug Smythies (dsmythies)]

The installer stuff is pretty obsolete and nobody seems to maintain it anymore. It was suggested to not even have it on the 13.04 help pages (it isn't included on the 12.10 help pages) However, and due to the many links that would become broken, I put it there and just changed the titles away from "alternate CD".
Anyway, the correct package is "installation-guide" and I will change it in a moment.

[Nathan Dorfman (ndorf)]

Nevermind, I see the GUI creator tool now. FWIW, this way had a nice advantage -- instead of overwriting the stick each time, you could have as many ISOs and installers as you wanted, all on the existing vfat filesystem.

[Quinn Balazs (qbalazs)]

If they're kept, I'd be more than willing to work on getting the install pages up to date.

[Doug Smythies (dsmythies)]

@Quinn: There is pressure to delete the installation guide from the 13.10 pages (preliminary pending) of help.ubuntu.com. If you and/or others will work on it, then it can be argued to keep it on help.ubuntu.com. However, documentation string freeze is coming up soone (Sept 19th).

[Quinn Balazs (qbalazs)]

I'd argue that having some form of decently up to date install instructions is quite important. I'm pretty sure I can come up with a few people to help give this a shot. Creating up to date documentation would also limit the number of occurrences either in IRC, or here with people starting an install method and then having it fail when they perceived it to be a perfectly valid method. Install instructions however are notoriously difficult to keep up to date, as it requires pretty constant maintenance to ensure everything is still valid, the absence of which caused the ambiguity here. I feel with a few others, we could mostly if not entirely get it done before the betafreeze takes effect.

[Dimitri John Ledkov (xnox)]

Hm. to install from usb, one should simple dd the .iso onto the usb stick.
The hd-media methods should only work with alternate, server, mini .iso.
All of which can simply be dd onto the usb stick as well...

From the download pages on the ubuntu.com website we do clearly show and explain how to boot from USB:
http://www.ubuntu.com/download/desktop/install-desktop-long-term-support

See links on the right hand side.

[Nathan Dorfman (ndorf)]

Dmitrijs-- it's not as simple as you make it out to be. dding an image
over the USB stick requires a stick dedicated to the install, since it
will overwrite whatever filesystem was there before. The hd-media
installer on the other hand lets you just add/remove ISO files to the
existing filesystem at will, without overwriting anything.

[Dimitri John Ledkov (xnox)]

On 12 September 2013 15:27, Nathan Dorfman <email address hidden> wrote:
> Dmitrijs-- it's not as simple as you make it out to be. dding an image
> over the USB stick requires a stick dedicated to the install, since it
> will overwrite whatever filesystem was there before. The hd-media
> installer on the other hand lets you just add/remove ISO files to the
> existing filesystem at will, without overwriting anything.
>

Correct, but I doubt hd-media method works with "live" based images
(e.g. all desktop-live cd images with ubiquity based installer).

Regards,

Dmitrijs.

[Nathan Dorfman (ndorf)]

Oh, we all agree that it doesn't work. :) I may have misread your
comment, but it sounded to me like you were saying "the dd method is
just as good," which as I see it would just not be correct. Even if we
(reasonably) conclude that one of the methods isn't worth supporting
because of simple lack of interest, I think it'd be worth having the
drawbacks on the record -- you never know when someone might show up
willing and able to work on them.

BTW, it never occurred to me to try the server ISO. I guess it may
very well still work! I will give it a shot and report back.

[Quinn Balazs (qbalazs)]

In working on the installation guide, something that I noticed was that there is not a rundown of the "pros and cons' of each installation type. With the go-ahead from the installer team, I would like to add such a section to the guide. Having a section like this will make the decision of which install type will work best with the user's desires and the constraints of their hardware.

[Nathan Dorfman (ndorf)]

It turns out that there's an even better way to install all versions of Ubuntu, including 13.10, using a USB stick. It's easy, doesn't affect any existing files on the USB drive, and offers maximum flexibility at boot using grub. In my opinion it should be the first and foremost recommended installation method. Here's what I just did to install saucy on my previous raring machine:

1. Run the following command as root. This will install the bootloader and create a /boot/grub directory on the drive, but won't touch any other files. Very little space is required.

    grub-install --no-floppy --boot-directory=/media/$USER/$DISK_NAME/boot /dev/$DISK_DEVICE

2. Copy the Ubuntu installation ISO to a location of your choice on the USB drive, for instance /isos. If you want to be able to boot other versions of Ubuntu from the same drive, copy the corresponding ISOs as well.

3. Create a /boot/grub/grub.cfg file on the drive, containing a menuentry like the following for each ISO you want to boot:

    set timeout=10
    set default=0

    menuentry "Ubuntu 13.10 Saucy amd64" {
        set iso=/isos/ubuntu-13.10-desktop-amd64.iso
        loopback loop $iso
        linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$iso splash --
        initrd (loop)/casper/initrd.lz
   }

(If you have only one menuentry, you may want to set the timeout to 1-3 or even 0.)

That's it, you can now boot from the USB stick. This was much cleaner than the usb-creator-gtk tool, which unpacked the ISO into the USB drive's root directory. It also segfaulted about 5 times before successfully completing, while this method worked on the first attempt :)