iSCSI installation on root too complicated

Bug #447377 reported by sub.mesa on 2009-10-09
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
netbase (Ubuntu)
Undecided
Unassigned
partman-iscsi (Ubuntu)
Undecided
Unassigned

Bug Description

Binary package hint: partman-iscsi

Now that the server daily karmic builds support installing to iSCSI targets with no local disks present, this makes diskless ubuntu systems possible. While the installation itself was very easy, setting up the network boot correctly afterwards is not. Both conflicting and outdated information on the web leads to confusion, and no official documentation exists to get this thing working.

Basically, to install Ubuntu to iSCSI the general order of setup is:
- Setup DHCP, TFTP, iSCSI-target on the server OS (in my case this is FreeNAS)
- Boot from Ubuntu server cd, during the partitioner select "Login to iSCSI target" or something similar
- Installation will finish and it would want to reboot
- After rebooting, there is no local disk to boot from. So the boot will fail.
- After setting up DHCP and TFTP, the network boot works and Ubuntu kernel gets loaded
- The Ubuntu kernel needs access to the network interface and iSCSI-enabled initial ramdisk (initrd), to allow access to the iSCSI root device which Ubuntu has been installed to. This is where i am currently stuck.
- After mounting root, Ubuntu can continue to boot. I haven't reached this step yet.

So, the issue is with having a proper initrd file that supports iSCSI, and having the correct contents of the pxelinux.cfg file together with DHCP root-path data.

Perhaps this whole procedure would be alot more easy if:
1) ubuntu has an initrd-iscsi file somewhere that's compatible with my current kernel/distribution
2) there are clear instructions/documentations on how to properly setup initrd yourself
3) the installer configures the initrd properly so you don't need to do it; it should also prompt to upload this file or some other way to get it on your TFTP server.

For example, after installation to iSCSI target is successful, it can help the user alot by automating things needed to make the booting work. Right now all it does is install to the iSCSI device but it won't ever get the chance to access this information again -- unless the TFTP/DHCP services are correctly setup. Perhaps a sample compressed archive containing a basic working TFTP setup could be presented to the user. This file could be sent to the TFTP server, extracted, and the network boot thing should work out of the box. This is much more user friendly than the current installation. At least - i won't get this thing working without help. If that's the case there are likely more people like me.

That said iSCSI on root is rather new for Ubuntu, i really hope we can make this feature work with less problems/effort.

More info, previous bug:
https://bugs.launchpad.net/ubuntu/+source/partman-iscsi/+bug/435290

Forum topic containing contents of DHCP/TFTP setup:
http://ubuntuforums.org/showthread.php?p=8074893#post8074893

Colin Watson (cjwatson) wrote :

Can you check whether this works in 9.10? This sounds a bit like something that might have been caused by bug 457767, and I'd like to find out what's still broken.

sub.mesa (sub-mesa) wrote :
Download full text (4.6 KiB)

I tried this with 9.10 server amd64 final ISO, but the problems stay:

=========

PROBLEM #1: no suitable initrd.img file is provided that allows for iSCSI-on-root booting, though it is possible to create one after installing 9.10 to a VM for example and create a custom initrd.img file, like i did.

mkdir ~/tmp
cd ~/tmp
gunzip < /boot/initrd.img-2.6.31-14-server | cpio -i --make-directories

touch etc/iscsi.initramfs
cp -p /lib/modules/2.6.31-14-server/kernel/crypto/crc32c.ko lib/modules/2.6.31-14-server/kernel/crypto/
cp -p /lib/modules/2.6.31-14-server/kernel/lib/libcrc32c.ko lib/modules/2.6.31-14-server/kernel/lib/
cp -p /sbin/iscsid sbin/
cp -p /sbin/iscsistart sbin/

find ./ | cpio -H newc -o > ~/initrd-iscsi.cpio
gzip ~/initrd-iscsi.cpio
mv ~/initrd-iscsi.cpio.gz ~/initrd-iscsi.img
// now move the initrd-iscsi.img in your home directory to the TFTP server, and make sure it is used by pxelinux.cfg config file

note 1) this procedure has to be performed each time the initrd.img file is updated by the Ubuntu update system; otherwise it would not receive the updates the update system thinks its making. The local /boot/initrd.img file would not be used; only the initrd.img file provided by the TFTP server is used.

note 2) the etc/iscsi.initramfs file needs to exist, even though being an empty file. Not creating this file would cause the pxelinux.cfg iscsi login data to not be used. If you prefer getting the iSCSI authentication data from pxelinux.cfg file like i did, simply create an empty file by using the touch command in the above example. Putting the authentication data in the /etc/iscsi.initramfs file instead might be more secure, however. But only barrely as this is unencrypted and located on the TFTP server as well; which does not provide for any authentication by itself.

=========

PROBLEM #2: after having a suitable initrd.img file, iSCSI booting works. But upon rebooting or shutting down, the rc.d shutdown script configuration causes a bug that shuts down networking before the (iscsi-mounted) root filesytem is unmounted, causing an unclean shutdown with possible dataloss/corruption. It also requires a manual reset to continue. To fix this, i used the following commands:

cd /etc/rc0.d/
rm S35networking
mv S41open-iscsi S80open-iscsi
cd /etc/rc6.d/
rm S35networking
mv S41open-iscsi S80open-iscsi

Note 1) the file S41open-iscsi may be called differently; i do not remember 100% correctly; but anyway the open-iscsi file in the /etc/rc0.d/ and /etc/rc6.d/ directories have to called S80open-iscsi to allow clean shutdown
Note 2) removing the S35networking may have side effects i'm unaware of; it does seem to work without issues for me. I don't really understand why the network interface has to be "down" before shutting down anyway. The only thing important is that the root filesystem is unmounted cleanly, and the network interface has to be up long enough to make that happen. Removing the network shutdown script seemed the most simple solution to me.

=========

After coping with these two problems, diskless Ubuntu systems work. To get a desktop, i used the following commands:
sudo apt-get install ubuntu-desktop
reboot

Hope this helps ...

Read more...

Colin Watson (cjwatson) wrote :

Could you tell me whether /etc/iscsi/iscsi.initramfs exists, and if so what it contains? Furthermore, could you attach /var/log/installer/syslog?

Changed in partman-iscsi (Ubuntu):
status: New → Invalid
status: Invalid → Incomplete
Colin Watson (cjwatson) wrote :

By the way, the reason I'm asking these questions is that the installer is *already* supposed to set things up such that the initrd is generated with iSCSI support if the root filesystem is on iSCSI. (We don't do it by default because we have very aggressive boot time targets and every additional chunk in the initramfs slows things down; in any case, there's no point putting iSCSI support in the initramfs unless the configuration is also in place, and you're seeing this bug precisely because the configuration is not in place for some reason.) You're phrasing your comments in terms of a feature request; what I'm saying is that the feature is already present, but for some reason it is not working for you, hence my requests for specific files.

I agree that the networking setup is wrong. The least invasive fix for this will be to extend the check for network filesystems that's already in that script to detect root-on-iSCSI as well.

Could you file a separate bug report with your TFTP suggestions, please? We'd have to think about them in some more depth, and they're not specific to iSCSI.

Sadly, iSCSI support currently won't fit on the desktop CD, and is unlikely to be higher priority than other things the desktop team want to do ...

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netbase - 4.35ubuntu3

---------------
netbase (4.35ubuntu3) lucid; urgency=low

  * debian/netbase.init:
    - Don't deconfigure network interfaces if /etc/iscsi/iscsi.initramfs
      exists, since that probably indicates root on iSCSI (LP: #447377).
 -- Colin Watson <email address hidden> Thu, 03 Dec 2009 16:05:30 +0000

Changed in netbase (Ubuntu):
status: New → Fix Released
Colin Watson (cjwatson) wrote :

sub.mesa: Any progress on getting me the files with more information that I requested earlier? I'd like to make sure that this is cleaned up for Lucid, but I will need that information to do so. Thanks in advance.

eymey (eymert) wrote :

@Colin: I don't know if sub.mesa already has solved this, but I encountered just the same frustration and problems als he did :-)

I think the most frustrating part is, that there is no documentation or any note in the installer regarding iSCSI on root installations, other than that it CAN and DOES install the system on an iSCSI-based root filesystem, but no instructions or actions on where to find an initramfs image that can be used to make mounting the iSCSI volume doing the boot process possible :). In other words: The Ubuntu 9,10 release notes state explicitly iSCSI support has been improved and installation of root on iSCSI is possible, but it is practically not useable without a lot of hassle.

In an earlier response to this bug, you mention a file called "/etc/iscsi/iscsi.initramfs". When I get home and have the time (but ASAP) I will see if this file exists on the newly-installed system .Is this file useable as a ready to use initrd or do you still have to do something to create one? But even if it exists, there still is no easy way to get this distributed to a boot medium.

Scenario:
* Instantiate a Virtual Machine (eg. in VirtualBox) with no local storage attached
* Boot the Ubuntu server 9.10 ISO
* During installation, login to an iSCSI target and let the system install to it.
* After installation, the system wants to reboot

This is where the problem start: The system reboots and you have no way of reaching any initramfs, other than mounting the root filesystem on the iSCSI target from another system and (worst case) chroot to it if creating an initrd is still needed.

In my opinion (and I think many people will agree), the installation scenario should continue:

* After installation, an extra step is performed, in which the system will create the right initrd needed to boot this particular system (with the modules of the NIC that was in use during iSCSI login for installation, with the iSCSI modules and login procedure, anything to be able to mount the root FS).
* An option is given to SCP the newly created initrd and kernel to another system, or even put it on a USB key.

Any thoughts on this? :-). I am happy to test any howto/procedure you can give me, or a pointer to a place sub.mesa or I might not have searched for documentation.

eymey (eymert) wrote :

@Colin Watson:

OK, just looked up the "etc/iscsi/iscsi.initramfs" file and it only shows 4 lines containing the details of my iSCSI target (target name, target IP, etc.).

Also, the initramfs configuration does not show anything at all that might indicate instructions to the mkinitramfs command to include the right iSCSI setup procedure in the initrd-stage.... :(

I also included the installer syslog :)

sub.mesa (sub-mesa) wrote :
Download full text (4.6 KiB)

Colin, my apologies for my late response. I found some time to play with iSCSI again and used a fresh setup with final Ubuntu 9.10 server, without much issues.

The problems i had appear to originate from the date Ubuntu 9.10 was still in RC-phase; even after upgrading them as the final came out. I just setup a completely new Ubuntu 9.10 server final iscsi install and the initrd.img and vmlinuz written to the iscsi filesystem are indeed usable and work for me. The problem as eymey explained, is getting that initrd.img and vmlinuz (2 files) to the TFTP server used for booting. I'll file a separate bug on that, as you requested.

I simply used an initrd i already had to boot the system, then copied the /vmlinuz and /initrd.img files (which are symlinks and kept up-to-date to the currently installed kernel/initrd.img) to the TFTP server which pxelinux reads before the kernel is even started. An interum workaround would be to put the initial two files of a blank installation somewhere on the ftp server. I don't know about the 'dynamic' creation of that /etc/iscsi/iscsi.initramfs file though. I booted with an initrd from a different system, which would need to have different data in that file, and it still worked. Possibly because its dynamically generated when booting as it gets the data from pxelinux.cfg file which gives all iscsi information, snippet:

  APPEND ip=bootp root=/dev/sda1 iscsi_initiator=X iscsi_target_name=X iscsi_target_ip=X iscsi_target_group=1 iscsi_target_port=3260 iscsi_username=X iscsi_password=X initrd=/tftp/initrd.img
(removed actual data by X for privacy)

I believe this data is used to generate the /etc/iscsi/iscsi.initramfs file dynamicaly/automatically when the script activates upon boot.

I then upgraded using sudo apt-get update followed by sudo apt-get dist-upgrade, and could copy the 2 files to my remote server again, and booting worked fine. I created a small script i could run everytime after installing ubuntu updates that involve kernel/initrd changes. I simply run ~/updateiscsi and its copied to my remote server. So, i do not have to generate an initrd each time after upgrading; thus the auto-generated initrd.img is fine when using ubuntu 9.10 final.

So while it works, there are lots of rough edges:
- How to initially boot after installing; you need the initrd.img and vmlinuz files on the iscsi system but you can't get to that easily.
- After updating you need to manually synchronise your vmlinuz and initrd.img files to the TFTP server used for booting
- /etc/rc0.d and /etc/rc6.d hacks are still required for clean shutdown on iscsi-on-root systems
- /etc/network/interfaces must be manually created and set to static ip configuration; it doesn't appear to pickup the gateway information from the DHCP server initally
- If any local disks are present which you may or may not want to use as additional storage - not as root/system filesystem for Ubuntu - you need to change the pxelinux.cfg file to set root=/dev/sdb1 (for 1 local disk present) or to root=/dev/sdc1 (2 local disks present). Using UUIDs here probably won't work. Ideally, we would want a script that automatically checks which of the /dev/ entries is an i...

Read more...

mercedes (preciousa1237) on 2010-03-03
Changed in partman-iscsi (Ubuntu):
status: Incomplete → Fix Released
Colin Watson (cjwatson) wrote :

mercedes, please don't close bugs (a) without comment (b) without discussion with the responsible developer (c) when they're not actually really all-the-way fixed ...

Changed in partman-iscsi (Ubuntu):
status: Fix Released → Triaged
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers