iSCSI install fails under hardy

[Kevin Otte]

Steps performed:

- Booted with Hardy (8.04) Server CD
- Issued command line "install iscsi=true"
- Followed prompts to Configure iSCSI.
- Entered IP address of iSCSI target

Expected result
- Presentation of targets on given host to select from and continue the install

Actual result
- Red screen: "An installation step failed. ... The failing step is: Configure iSCSI"

Additional info / Troubleshooting steps:
A quick glance at the syslog output on tty4 indicates that the iSCSI subsystem failed to start because the InitiatorName was not set in /etc/iscsi/initiatorname.iscsi. Examination of this file via tty2 verifies this is the case. Its contents are a bunch of comment lines followed by:

InitiatorName=

I attempted to set this value with nano and re-perform the Configure iSCSI step. This was unsuccessful as my changes to /etc/iscsi/initiatorname.iscsi were overwritten by d-i.

Hardware issues have been ruled out as I have attempted this on three different physical machines.

[Matheos (matheos)]

Installer package open-iscsi-udeb has setting GenerateName=yes.
It's ok but uses iscsi-iname which it cannot find to autogenerate...
It invokes /usr/bin/iscsi-iname - proper location is /sbin/iscsi-iname

quick workaround: make symbolic link before starting iscsi

ln -s /sbin/iscsi-iname /usr/sbin/iscsi-iname

[Etienne Goyer (etienne-goyer-outlands)]

Confirmed using 8.04.1 server CD; using the ln trick above actually let me go through the installer.

Moreover, no record are being created for the iscsi target defined during installation, and thus boot fail (by dropping to a root maintenance shell) right after installation as the iscsi volume cannot be mounted. It is easy enough to fix by running "sudo iscsiadm -m discovery -t st -p <ip of target>", but I presume something went wrong during the installation. Otherwise, the record would exist.

[Etienne Goyer (etienne-goyer-outlands)]

I take the last part back: running "iscsiadm -m discovery -t st ..." does not fix the problem at boot. Sorry for the misleading comment. Plus, it is only tangentially related to the actual bug that is the topic of this report; I should file another one.

[Etienne Goyer (etienne-goyer-outlands)]

Also confirmed on Intrepid, using the beta server CD (i386).

[Steve Langasek (vorlon)]

Accepted the nomination for intrepid, but of course if we don't squeeze this in before final it will be 'wontfix' since there are no other CD rebuilds for 8.10 after release.

[Etienne Goyer (etienne-goyer-outlands)]

Can we nominate that one for 8.04.2? Launchpad does not let me do it.

[Kevin Otte (nivex)]

Any chance we'll see this in Jaunty? Jaunty+1? What is the timeline for this?

[Kevin Otte (nivex)]

iscsi=true doesn't even seem to be parsed by the Jaunty alpha4 installer.

[Colin Watson (cjwatson)]

iscsi=true should still be supported in Jaunty; I can find no evidence that it has been removed from the installer. preseed_aliases still lists it and disk-detect.sh still handles it.

[Colin Watson (cjwatson)]

I've uploaded a partial fix (which needs release team approval since we're in freeze), although I can't test this myself so there may well be more to it than this. Still, this part by itself seems obviously correct to me.

open-iscsi (2.0.870.1-0ubuntu3) jaunty; urgency=low

  * Invoke iscsi-iname using normal $PATH lookup rather than using an
    incorrect explicit path (see LP #236640).

 -- Colin Watson <email address hidden> Thu, 09 Apr 2009 16:49:08 +0100

[Colin Watson (cjwatson)]

Could somebody please test a current daily build with iscsi=true to see how it behaves, and post logs if it breaks? My fix was accepted, but I can't easily test this myself.

[malaiwah (michel-belleau)]

Colin, did you also check for the "awk" requirement of the "/sbin/iscsi_discovery" script?
I'll try a daily build later today, but the latest Beta from Jaunty was having this problem.

I successfully started installation of Jaunty on a OpenSolaris Zvol (iSCSI target) using these three commands at the console (ALT-F2) before entering my target hostname:

udpkg -i /cdrom/pool/main/g/gawk/gawk*.deb
ln -s /usr/bin/gawk /bin/awk
ln -s /sbin/iscsi-iname /usr/sbin/iscsi-iname

By the way, iscsi=true is correctly parsed by the Jaunty installer; just make sure you are not concatenating it with "--" which is at the end of the default boot command line.

[malaiwah (michel-belleau)]

Using the daily build, I can't even get to the iSCSI target dialog: it stops at the "Detect disks" action.

Here is what I found looking at the console log (sorry cannot copy/paste at this time):
<<<
anna-install: Installing open-iscsi-udeb
main-menu: unknown udeb open-iscsi-udeb
main-menu: WARNING Configuring 'disk-select' failed with error code 1
main-menu: WARNING Menu item 'disk-select' failed
>>>

[malaiwah (michel-belleau)]

Running "find" on "/cdrom/pool" mentions no entries about "iscsi"; that package does not seem to be included in the daily build CD.

[Steve Langasek (vorlon)]

iscsi is included only in the server CD builds; it is there in the Ubuntu 9.04 server release.

Can anyone provide the logs requested by Colin in <https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/236640/comments/11>?

[Kevin Otte (nivex)]

Steve, there was a response to comment 11 in comment 13. I can't seem to find where to get the daily build (my google-fu is weak today)

I just tried the 9.04 server release CD with iscsi=true and was greeted with the dialog asking me to choose a module for my storage device. I grepped syslog for iscsi and the only mention was the boot command line. open-iscsi deb and udeb are present in the pool, but they don't appear to be loaded by anything.

[Dustin Kirkland  (kirkland)]

Kevin-

Daily builds are available at http://cdimage.ubuntu.com.

:-Dustin

[Colin Watson (cjwatson)]

Kevin: comment 13 is apparently from an alternate install CD; that message can only be explained by running off a CD that doesn't include open-iscsi-udeb, and the server CD definitely does contain that package. As such, I'm still waiting for feedback on this bug.

If at all possible, I need the *full syslog*. Please don't try to cut it down to just the bits that look relevant - it's much easier to work with the whole thing. You can go back to the main menu and use the "Save debug logs" item to extract it, or you can type 'anna-install openssh-client-udeb' on tty2 and then scp /var/log/syslog to another machine.

[Colin Watson (cjwatson)]

Please also state the URL from which you downloaded the CD image in any report.

[Etienne Goyer (etienne-goyer-outlands)]

There is no daily of the server flavour on http://cdimage.ubuntu.com, but here is /var/log/syslog from Jaunty server ISO, when launched with "iscsi=true".

Steve, contrary to what we discussed earlier today, I do think finally it is the same error we get on jaunty and hardy after all. It does appear that awk is called from /sbin/iscsi_discovery, but is missing from the installer. Since open-iscsi is still version 2.0.870.1-0ubuntu3 in karmic (same as jaunty), I can only presume karmic is also affected.

[Dustin Kirkland  (kirkland)]

On Thu, Jun 25, 2009 at 4:21 PM, Etienne
Goyer<email address hidden> wrote:
> There is no daily of the server flavour on http://cdimage.ubuntu.com,

Etienne:

Is this what you're looking for?
http://cdimage.ubuntu.com/ubuntu-server/daily/current/

:-Dustin

[Colin Watson (cjwatson)]

On Thu, Jun 25, 2009 at 09:53:59PM -0000, Dustin Kirkland wrote:
> Is this what you're looking for?
> http://cdimage.ubuntu.com/ubuntu-server/daily/current/

Unfortunately broken right now due to a busybox bug - should have it
fixed soon.

[Steve Langasek (vorlon)]

From what I see, the hardy version of iscsi_discovery doesn't use awk at all; so that aspect of this bug seems to only apply to later releases, and fixing the iscsi-iname invocation, as mentioned in comment #2, looks like it should be sufficient for hardy. Preparing an SRU upload.

[Steve Langasek (vorlon)]

SRU uploaded and awaiting approval; looking now at fixing iscsi_discovery in karmic to not depend on awk.

[Steve Langasek (vorlon)]

Attached is a debdiff for karmic that removes the dependency on awk. I've tested the first codepath, but not the second because I don't have access to an iscsi server; I would appreciate it if someone could test iscsi_discovery (in an installed system, not the installer) before I upload this.

[Martin Pitt (pitti)]

Accepted into hardy-proposed. Please test.

[Etienne Goyer (etienne-goyer-outlands)]

Attached is /var/log/syslog from the daily karmic server ISO (June 29th).

However, it does not include the fix Steve uploaded in comment 25 (open-iscsi 2.0.865-1ubuntu4). As it is an installer bug, I guess it will ned to find it's way into the daily build before I can test.

[Etienne Goyer (etienne-goyer-outlands)]

As I explained a couple months earlier (see comment 2), even if the path to iscsi-iname is fixed, it still fail in the end. The installer goes on just fine, but if you choose to configure a file system residing on an iSCSI target to be mounted at boot time (not as the root file system, just about any mount point), /etc/rcS.d/S30checkfs.sh will fail at boot and drop to a maintenance shell.

This is because the iSCSI block device is not brought up on boot correctly, and as such the file system residing on that iSCSI target cannot be checked/mounted. For it to be so, there need to be a valid node configuration file under /etc/iscsi/nodes/. This configuration file is being generated just fine by the open-iscsi udeb in the installer environment. However, it is *not* being copied to the root file system of the installation (/target). There are two problems:

1. The post-installation script of the udeb (debian/open-iscsi-udeb.finish-install in the source deb) is being installed in /lib/finish-install.d/. I think this is supposed to be /usr/lib/finish-install.d/.

2. There is a bug in debian/open-iscsi-udeb.finish-install. It copy the whole configuration directory with:

    cp -a /etc/iscsi /target/etc/iscsi

Which mean we end up with the installer iscsi configuration directory as /etc/iscsi/iscsi in the target, which is not what we want. I presume it should really be:

    cp -a /etc/iscsi /target/etc/

So that we have the configuration from the installer environment copied over to the target properly.

The above problems where found on both hardy and jaunty.

Moreover, there seems to be a race condition at boot time where open-iscsi is started before networking is ready, which also prevent the iSCSI block device to be available when /etc/rcS.d/S30checkfs.sh is being run. I am still investigating and will report a separate bug as appropriate.

[Steve Langasek (vorlon)]

On Mon, Jun 29, 2009 at 02:28:08PM -0000, Etienne Goyer wrote:
> However, it does not include the fix Steve uploaded in comment 25 (open-
> iscsi 2.0.865-1ubuntu4). As it is an installer bug, I guess it will ned
> to find it's way into the daily build before I can test.

It has not been uploaded. I'm still waiting for feedback that this works
correctly in an installed system before I upload this untested change.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Etienne Goyer (etienne-goyer-outlands)]

Steve, I built open-iscsi with the above debdiff, and iscsi_discovery does work just like the unpatched version. I found two other bugs with it (the block device is not being made available right away, and it cause a kernel oops in certain circumstance), but I will be reporting these separately.

[Etienne Goyer (etienne-goyer-outlands)]

The kernel oops have been reported as bug #395323.

The target block device not being made available right away is not a bug, it is the expected behavior. Sorry for the noise.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-iscsi - 2.0.870.1-0ubuntu5

---------------
open-iscsi (2.0.870.1-0ubuntu5) karmic; urgency=low

  * utils/iscsi_discovery: replace uses of awk with other shell voodoo, so
    that this works in the installer. LP: #236640.
  * debian/rules, debian/open-iscsi-udeb.dirs: install the open-isci-udeb
    finish script to /usr/lib/finish-install.d, not to /lib/finish-install.d.
  * debian/open-iscsi-udeb.finish-install: copy the config from the root
    to the correct directory in the target.

 -- Steve Langasek <email address hidden> Tue, 07 Jul 2009 22:10:17 -0700

[Steve Langasek (vorlon)]

Etienne,

I'm preparing an SRU upload of open-iscsi for hardy including the latest fixes that you've indicated, which I'll be uploading today. In the meantime, would it be possible for you to do a full install test with the latest karmic daily, which includes the fixes from open-iscsi 2.0.870.1-0ubuntu5, to confirm that this addresses the problem there?

[Kevin Otte (nivex)]

I just acquired karmic-server-i386.iso from http://cdimage.ubuntu.com/ubuntu-server/daily/current/ and booted with the iscsi=true commandline option. I was not prompted for any iscsi information, nor is "Configure iSCSI" present in the installer main menu.

[Etienne Goyer (etienne-goyer-outlands)]

I do not have the problem Kevin reports above; iscsi=true works just fine for me. I tested using the karmic server CD image dated 08-Jul-2009 10:32 from the same location he does.

I am pleased to report the iSCSI installer support works just fine now for me. The iSCSI dialog come up fine, the partitioner present the iSCSI target as a valid block device, and configuring a file system on said iSCSI target to be mounted on boot work as expected. It sounds like the last upload actually fixed the bug!

[Colin Watson (cjwatson)]

Accepted into hardy-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Etienne Goyer (etienne-goyer-outlands)]

Colin, as the bug is in the installer part of the package, I would need an ISO to test. Or can I test using the netboot installer?

[Colin Watson (cjwatson)]

open-iscsi-udeb isn't in the installer initrd, but instead is retrieved from the archive at run-time; so booting the installer with apt-setup/proposed=true (once this has built and published) should be sufficient.

[Colin Watson (cjwatson)]

(err, yes, obviously that approach only works with netboot not CD)

[Steve Langasek (vorlon)]

On Wed, Jul 08, 2009 at 10:52:30PM -0000, Etienne Goyer wrote:
> Colin, as the bug is in the installer part of the package, I would need
> an ISO to test. Or can I test using the netboot installer?

The ubuntu-server hardy daily builds for 20090709 include the updated
open-iscsi-udeb:

  http://cdimage.ubuntu.com/ubuntu-server/hardy/daily/20090709/

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Etienne Goyer (etienne-goyer-outlands)]

The latest daily hardy server CD ISO (20090709) works just fine indeed. When provided with the iscsi=true boot option, it will prompt for the iSCSI target to use, the iSCSI target will show up as a block device in the partitionner dialog, and you can use it for non-root file systems just fine. I used the iSCSI target to host the /home file system, and it worked fine.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-iscsi - 2.0.865-1ubuntu3.3

---------------
open-iscsi (2.0.865-1ubuntu3.3) hardy-proposed; urgency=low

  * Backport from karmic:
    - debian/rules, debian/open-iscsi-udeb.dirs: install the open-isci-udeb
      finish script to /usr/lib/finish-install.d, not to
      /lib/finish-install.d.
    - debian/open-iscsi-udeb.finish-install: copy the config from the root
      to the correct directory in the target.
    - debian/open-iscsi-udeb.start: invoke iscsi-iname using normal
      $PATH lookup rather than using an incorrect explicit path.
    LP: #236640.

 -- Steve Langasek <email address hidden> Sat, 27 Jun 2009 14:22:23 -0700

[Kevin Otte (nivex)]

When I was testing I had forgotten that there had to be a local disk present for open-iscsi to get loaded. I retested this morning with a dummy hard disk in play and the install completed successfully. Thank you all for your hard work on this one!

[sub.mesa (sub-mesa)]

Sorry for bumping this bug, but i files a similar bug here:
https://bugs.launchpad.net/ubuntu/+bug/435290

Reading the last statement that there has to be a local disk present for the iSCSI package to become loaded; is this intentional or is this a bug? It surely isn't consistent with the goals of something using iSCSI; to have a diskless ubuntu client. So why then is it a requirement to have a local disk present?

Why doesn't Ubuntu display a friendly message when no local disks are found, with a few options such as activating support for iSCSI to install to the network instead; without any local disk present. This is how iSCSI should be used, when installing Ubuntu to, IMO.

[Etienne Goyer (etienne-goyer-outlands)]

sub.mesa: this is a different issue. We did not had root file system over iSCSI support in hardy at all (it is just shaping up for karmic).