Ubuntu 16.04: netinstall/boot installation files not found on ppc64el ISOs

[bugproxy]

== Comment: #0 - Kalpana Shetty <email address hidden> - 2015-01-05 04:27:08 ==
---Problem Description---
Ubuntu 15.04: netinstall/boot installation files not found for Ubuntu 15.04

Contact Information = <email address hidden>

---uname output---
ubuntu 15.04 kernel

Machine Type = Power 8

---Debugger---
A debugger is not configured

---Steps to Reproduce---
netinstall files are not found at below location for Ubuntu 15.04:
http://cdimage.ubuntu.com/netboot

Please upload the netinstall files for Ubuntu 15.04.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1407714/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-01-15 18:18 EDT-------
Hi Ubuntu,

We have noticed that the powerpc and ppc64el server ISO images do not include the netboot kernel/initrd, as do the amd64 ISOs. There are other kernel/initd on the disc but they are apparently not suitable for netboot (i.e.. missing network drivers) and so we need to bring in the netboot files from some other source (download).

This is also a issue for mgmt systems such as xCAT, which typically set up netboot by pulling the kernel/initrd off the ISO. So, xCAT can do this for amd64, but not for ppc.

Is there a reason why this is? If not, we would like to have the netboot kernel/initrd included on the ISO. Can we do this?

[Adam Conrad (adconrad)]

The d-i netboot images for ppc64el are published here:

http://ports.ubuntu.com/ubuntu-ports/dists/vivid/main/installer-ppc64el/

However, I'll update the friendly links from cdimage/netboot to point to the right places for vivid right now, for people who don't know all the deep links.

[Adam Conrad (adconrad)]

http://cdimage.ubuntu.com/netboot/ has been updated with pointers to the right bits for vivid.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-01-16 23:12 EDT-------
(In reply to comment #8)
> The d-i netboot images for ppc64el are published here:
>
> http://ports.ubuntu.com/ubuntu-ports/dists/vivid/main/installer-ppc64el/
>
> However, I'll update the friendly links from cdimage/netboot to point to the
> right places for vivid right now, for people who don't know all the deep
> links.
>
> http://cdimage.ubuntu.com/netboot/ has been updated with pointers to the
> right bits for vivid.

I think Dave may not have been clear enough.

The issue isn't trying to find the netboot ISOs. We get those just fine by following the normal links at http://cdimage.ubuntu.com/netboot/vivid/:

e.g., to http://ports.ubuntu.com/ubuntu-ports/dists/vivid/main/installer-ppc64el/current/images/netboot/

The issue is that, the DVD iso: http://cdimage.ubuntu.com/ubuntu-server/daily/current/ on ppc64el does not allow for netbooting. It is missing some drivers in the initrd, but also even when said kernel/initrd is netboot'd, it attempts to install off the DVD. I have seen this for all ppc64el releases of Ubuntu when installed via Cobbler. A workaround (which I've been using for some time) is to use the vmlinux/initrd.gz from http://ports.ubuntu.com/ubuntu-ports/dists/vivid/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/ to override Cobbler's notion of Ubuntu's kernel/initrd pair.

From what Dave has told me offline, though, x86 doesn't suffer from this particular issue. The same ISO can be used to install from the disk tray or over the network.

Thanks,
Nish

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-01-21 18:25 EDT-------
(In reply to comment #9)
> (In reply to comment #8)
> I think Dave may not have been clear enough.
>
> The issue isn't trying to find the netboot ISOs. We get those just fine by
> following the normal links at http://cdimage.ubuntu.com/netboot/vivid/:
>
> e.g., to
> http://ports.ubuntu.com/ubuntu-ports/dists/vivid/main/installer-ppc64el/
> current/images/netboot/
>
> The issue is that, the DVD iso:
> http://cdimage.ubuntu.com/ubuntu-server/daily/current/ on ppc64el does not
> allow for netbooting. It is missing some drivers in the initrd, but also
> even when said kernel/initrd is netboot'd, it attempts to install off the
> DVD. I have seen this for all ppc64el releases of Ubuntu when installed via
> Cobbler. A workaround (which I've been using for some time) is to use the
> vmlinux/initrd.gz from
> http://ports.ubuntu.com/ubuntu-ports/dists/vivid/main/installer-ppc64el/
> current/images/netboot/ubuntu-installer/ppc64el/ to override Cobbler's
> notion of Ubuntu's kernel/initrd pair.
>
> From what Dave has told me offline, though, x86 doesn't suffer from this
> particular issue. The same ISO can be used to install from the disk tray or
> over the network.
>
> Thanks,
> Nish
>

Exactly. Here is a list of initrd files on amd64, powerpc and ppc64el ISOs, respectively. We notice the amd64 disc has an additional initrd specifically for netbooting, not present on the powerpc and ppc64el discs. The initrd on the powerpc and ppc64el discs are missing some or all of the required network drivers. On the ppc64el, I notice some of the drivers (e.g. ibmveth, tigon3) are built into the kernel and so do not need to be on the initrd. Even so, it's missing quite a lot. Like the powerpc64 case, there are a boatload of drivers in the "netboot" initrd that are not in these initrd.

$ find ./ubuntu15.04/ -name initrd.gz -exec ls -lh {} \;
-r--r--r-- 1 root root 9.9M Jan 13 01:46 ./ubuntu15.04/amd64/install/initrd.gz
-r--r--r-- 1 root root 22M Jan 6 20:10 ./ubuntu15.04/amd64/install/netboot/ubuntu-installer/amd64/initrd.gz
-r--r--r-- 1 root root 8.1M Jan 13 17:49 ./ubuntu15.04/ppc64/install/e500mc/initrd.gz
-r--r--r-- 1 root root 8.0M Jan 13 17:49 ./ubuntu15.04/ppc64/install/powerpc/initrd.gz
-r--r--r-- 1 root root 8.5M Jan 13 17:49 ./ubuntu15.04/ppc64/install/powerpc64/initrd.gz
-r--r--r-- 2 root root 9.3M Jan 13 17:49 ./ubuntu15.04/ppc64el/install/initrd.gz

(It's easy to see that the "netboot" initrd for PPC, not shown here, are about twice the size of the ones on the ISO).

So the question is, why not put the netboot initrd on the powerpc and ppc64el ISO as they are for amd64? If no good reason, can we get them added? Thx.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-01-23 20:36 EDT-------
I think today's issue points out why this is more than just an inconvenience: We have a new daily build ISO posted today but the netboot files are still from yesterday. (at the URLs mentioned by Nish above). So even though we have a new daily build we are blocked from netboot installing it until the new files are posted. Adding the netboot files to the ISO would prevent this. Thanks!

[Adam Conrad (adconrad)]

Err, the above comment makes no sense, FWIW. ISO build and debian-installer builds are not the same thing. In fact, the d-i on a new CD would still be the old one from days ago, even though the ISO would be new.

netbooting, by its very nature, pulls files directly from the archive, so the ISO is effectively worthless, and I'm not sure why you'd want to download something that's CD-sized when you only intend to use a few MB from it.

Anyhow, the reason to keep the powerpc ISO slim is fairly obvious, it contains several kernel flavours (which add up fast), so we try not to bloat it out too much.

The ppc64el ISO, however, having only one flavour, could probably have the netboot initrd added to it in the same fashion as we do for amd64. I'll look into that at some point.

Unless someone can give me a solid reason otherwise, however, I'm going to make this a wishlist priority, as the netboot kernel/initrd are published elsewhere (and those are exactly the same files that would be on the CD, feel free to validate that claim on amd64), so it's merely a convenience(?) to also let you download a whole CD to get them.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-01-26 23:14 EDT-------
(In reply to comment #16)
> Err, the above comment makes no sense, FWIW. ISO build and debian-installer
> builds are not the same thing. In fact, the d-i on a new CD would still be
> the old one from days ago, even though the ISO would be new.
>
Then this is an issue, when testing daily build ISOs, if the netboot files are not in sync with the ISO.

> netbooting, by its very nature, pulls files directly from the archive, so
> the ISO is effectively worthless, and I'm not sure why you'd want to
> download something that's CD-sized when you only intend to use a few MB from
> it.
>
The use case is netbooting to installing from a local (http or ftp) repository created from the ISO, not netbooting to installing from the internet repository..

> Anyhow, the reason to keep the powerpc ISO slim is fairly obvious, it
> contains several kernel flavours (which add up fast), so we try not to bloat
> it out too much.
>
So.. this would add three new initrd files, at about 20 mb each? assuming the old ones could not be replaced? If that's too much bloat, Ok.
>
> The ppc64el ISO, however, having only one flavour, could probably have the
> netboot initrd added to it in the same fashion as we do for amd64. I'll
> look into that at some point.
>
Cool.

> Unless someone can give me a solid reason otherwise, however, I'm going to
> make this a wishlist priority, as the netboot kernel/initrd are published
> elsewhere (and those are exactly the same files that would be on the CD,
> feel free to validate that claim on amd64), so it's merely a convenience(?)
> to also let you download a whole CD to get them.
>
Not sure why the request is so unreasonable. The benefit seem obvious, when you consider the use case of netbooting for installation against a local repository, which is a very common installation method in the enterprise. The pitfalls of not having them in the same place, and in sync, equally obvious. The netboot files are on the amd64 ISO so I'm not sure why the ppc64 and ppc64el cases are so different. Other leading distros put the netboot files on their server ISO as well, and many network mgmt systems (e.g. xCAT) are used to finding them there. In fact, the Ubuntu ppc ISOs are the first place I've hit this issue.

[Mauricio Faria de Oliveira (mfo)]

Hi Adam,

Thanks for looking into this.
I'd like add another reason for your consideration :)

> Unless someone can give me a solid reason otherwise, however, I'm going to make this a wishlist priority,
> as the netboot kernel/initrd are published elsewhere ([...]), so it's merely a convenience(?) to also let you download a whole CD to get them.

The biggest problem I've personally seen arises on 2 similar cases:
- some cluster/cloud deployments (e.g., xCAT)
- similarly, people/provisioning who use the netboot kernel/initrd and the ISO as HTTP mirror; either for
    - convenience; e.g., when there's no physical access to the systems; likely with POWER systems, AFAICT, usually on datacenters.
    - restriction; e.g., some scenarios/environments have no Internet access whatsoever, to reach Ubuntu mirrors.

All those people will eventually hit the following error (totally correct) in the installer, when the netboot kernel version they download becomes greater than what's in the ISO:
    'No kernel modules were found.
    This probably is due to a mismatch between the kernel used by this version of the installer
    and the kernel version available in the archive.'

.. because they can't access the new udebs w/ kernel modules in <distro>-updates repository (they're certainly not in the ISO)

It's also a problem because the older netboot files seems to be erased from time to time.. making someone who just picked the newly available netboot files not able to use ISO as mirror.

Since that limitation sorts of potentially limit a number of netboot+ISO deployments over time (when netboot kernel version jumps over available ISOs) -- which is true for 14.04 LTS point releases -- can this be increased from wishlist?

Thanks again!

P.S.:

I've personally had to deal with this, and the work-around of not downloading most of the -updates repo for the udebs and only getting what's needed included creating a fake -updates repository on the server, only for the udeb kernel modules on the version of the netboot kernel, and do d-i/allow_unauthenticated=true on the installation.

Please helps us not to suggest this :)
Not pretty, but here is goes:

Create fake -updates repo:

 # dir=/install/ubuntu14.04.1/ppc64el/

 # mirror='http://ports.ubuntu.com/ubuntu-ports'
 # dist='trusty-updates'
 # component='main'
 # arch='ppc64el'
 # path="$component/debian-installer/binary-$arch"

 # packages_file='Packages.gz'
 # packages="$dir/dists/$dist/$path/$packages_file"
 # release_file='Release'
 # release="$dir/dists/$dist/$release_file"

 # mkdir -p $dir/dists/$dist/$path

 # vmlinux="$dir/install/vmlinux"
 # version="$(grep 'Linux version' --text $vmlinux | cut -d' ' -f3)"

 # wget -O- "$mirror/dists/$dist/$path/$packages_file" \
   | gunzip --stdout \
   | sed -n -e "/^Package:/,/^\$/ H;" -e "/^\$/ { x; /Kernel-Version: $version/ s/\n//p }" \
   | gzip \
   > $packages

 # wget -O- $mirror/dists/$dist/$release_file \
   | sed -e '/:$/,$ d' -e "/^Components:/ s/: .*/: $component/" \
   > $release

 # size="$(wc --bytes < $packages | tr -d ' ')"
 # (
     cd $dir/dists/$dist
     file=$path/$packages_file
     echo 'MD5Sum:'
     md5sum $file
     echo 'SHA1:'
 ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-03-14 15:52 EDT-------
Even though we have this open for Vivid, AFAICT it affects all releases. This is the ugly URL where one must download the netboot files for Trusty 14.04.2:

http://ports.ubuntu.com/dists/trusty-updates/main/installer-ppc64el/current/images/utopic-netboot/ubuntu-installer/ppc64el/

[Steve Langasek (vorlon)]

> This is the ugly URL where one must download the netboot files for Trusty
> 14.04.2:

The recommended URL is in fact:

  http://cdimage.ubuntu.com/netboot/trusty/

[Zhao Er Tao (ertaozh)]

We also have encountered many issues when using the released ubuntu14.04.2 ISO and the netboot initrd.gz.
Only a specified netboot initrd.gz for our p8 tuleta-L machine, the latest initrd.gz from http://cdimage.ubuntu.com/netboot/trusty/ will cause networks issues.

[XiaoPeng Wang (daniceexi)]

We do think it's important to have corresponding network boot initrd packed in the release IOS, otherwise customers have to figure out workable initrd from a separated web site by themselves.

[ralph bellofatto (ralphbel)]

We have observed two problems here.

1. the initrd.gz that comes with the ISO image does not work for ppc64le.

2. The initrd.gz image at the time of filing this bug fails to recognize the broadcom network cards on the IBM tuletta ppc64le systems.

At one point during this year there was a 14.04.2 initrd.gz image that did work and we captured that and have kept a copy..

the file that was posted as of:

````
     ???????????????????? [!!] Configure the network ?????????????????????
     ? ?
     ? Installation step failed ?
     ? An installation step failed. You can try to run the failing item ?
     ? again from the menu, or skip it and choose something else. The ?
     ? failing step is: Configure the network ?
     ? ?
     ? <Continue> ?
     ? ?
     ?????????????????????????????????????????????????????????????????????

````

This copy obtained earlier this year works:
md5sum initrd.gz
61fc21cced97a2104ca3d768404b122a initrd.gz

However the copy obtained from:

http://ports.ubuntu.com/ubuntu-ports/dists/trusty-updates/main/installer-ppc64el/current/images/utopic-netboot/ubuntu-installer/ppc64el/initrd.gz

fails with the above error. When you go to the command shell and do an "ip a" you see that it failed to see any NIC cards.

Having a "working" copy of initrd.gz is very very important, and 2. not having to jump through hoops to deal with ppc64le architectures is also important.

Please get us a working copy of the initrd.gz and make it available on on the distro CD to make it easier for those doing network provisioning to set things up.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-06-25 16:23 EDT-------
(In reply to comment #29)
> We have observed two problems here.
>
> 1. the initrd.gz that comes with the ISO image does not work for ppc64le.
>
There *are* initrd files on the ISO but they are general not the netboot initrd, meaning simply that these initrd contain no (or, very few) network drivers. For the PPC archs, you need to download the netboot initrd from the upstream mirror.

The kernel image (vmlinux) appears to be identical to the one on the ISO, so there is no such thing as a "netboot kernel" as far a I know. There is a vmlinux file available for download at the same location as the netboot initrd, but it's identical to the one on the CD

> 2. The initrd.gz image at the time of filing this bug fails to recognize
> the broadcom network cards on the IBM tuletta ppc64le systems.
>
Assuming you have the netboot initrd... check to make sure this contains a working driver for your NIC. You can "execute a shell" from the installer and from there you can do lsmod, modinfo, etc. If you find a driver is missing or broken you can open a bug for that.

But in general, yes: it would be a heckuva lot easier if the netboot initrd was contained on the ISO.

[Mauricio Faria de Oliveira (mfo)]

> The kernel image (vmlinux) appears to be identical to the one on the ISO,
> so there is no such thing as a "netboot kernel" as far a I know.

The netboot kernel evolves/changes versions with the kernel package updates.
It causes a problem w/ the netboot installation from an GA ISO repository because the installer requires the kernel modules udebs at the same version of the running kernel.
If the netboot kernel is from an updated package, it eventually will have a different version number than the udebs at the ISO, and thus the netboot from ISO install will fail.

The meantime (long) workaround for that is described in LP bug 1407714 comment #10,
which is to create a fake trusty-updates repository with the udebs for the netboot kernel version, so the installer can find those, and proceed normally.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-12-03 06:13 EDT-------
Is there any update on this bug??

[Breno Leitão (breno-leitao)]

Adam,

if we get a chance to have this fixed in 16.04, we would appreciate. Xcat[1] uses the ISO image to get the initrd from, so, it would help users to use xcat to deploy Ubuntu.

[1] https://sourceforge.net/p/xcat/wiki/Main_Page/

[Mauricio Faria de Oliveira (mfo)]

I believe there are several other cases, specially among test teams and some cloud deployments, where they download an ISO and use it as a network install mirror, in order to save network bandwidth / install time.

However, they need to get the vmlinux/initrd from ports.ubuntu.com/netboot (with the network modules), and that quickly becomes out-of-sync with the kernel modules udeb packages from the ISO, rendering it unuseful.

If the change is not too much trouble, it would be really great to have this working.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-01-09 11:51 EDT-------
Making comment available externally as well:

(In reply to comment #44)
> (In reply to comment #34)
> I believe there are several other cases, specially among test teams and some
> cloud deployments, where they download an ISO and use it as a network
> install mirror, in order to save network bandwidth / install time.
>
> However, they need to get the vmlinux/initrd from ports.ubuntu.com/netboot
> (with the network modules), and that quickly becomes out-of-sync with the
> kernel modules udeb packages from the ISO, rendering it unuseful.
>
> If the change is not too much trouble, it would be really great to have this
> working.
>

To add support here is another example of such a use case:
OpenPOWER Cluster Genesis, a hardware provisioning tool that supports Ubuntu
installation (https://github.com/open-power-ref-design/cluster-genesis),
also utilizes the ppc64el netboot initrd along with the installation ISO
image. Some users require deployement within an isolated network without
access to an Ubuntu repository mirror. The x86 Ubuntu ISO images contain
everything needed to setup a local PXE boot environment (installation source
+ netboot initrd), while the ppc64el installation source and netboot initrd
must be obtained separately. A consistent, architecture independent approach
to setting up local Ubuntu PXE install environments would be more practical.

[Steve Langasek (vorlon)]

reopened ubuntu task for tracking purposes

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-01-19 21:25 EDT-------
Will there be an Ubuntu 14.04.6 LTS release? If so, please apply the fix to that release as well.

[Steve Langasek (vorlon)]

> Will there be an Ubuntu 14.04.6 LTS release?

No, there will not.

[Adam Conrad (adconrad)]

This should be fixed in the upcoming 16.04.2 RC image and in the next zesty daily.

[Joshua Powers (powersj)]

This appears fixed in the latest ppc64el ISO, however a consequence of this change is that the ISO has grown 60MB in size. This results in an ISO that is greater than 700MB, 704MB to be exact, meaning it cannot be copied or burned to a standard sized CD.