13.04 server: Fujitsu BX900/DX90 installation problem

[Andy Igoshin]

i try to install ubuntu 13.10 server on Fujitsu BX900 (BX924S3 diskless blade) whose storage is Fujitsu DX90 iSCSI target (iSCSI boot).

in the BX924S3 blade Emulex CNA 10G adapter is used. this device has hardware iSCSI support. i configure iSCSI initiators on it. after that corresponding LUNs become visible in the BX924S3 blade's BIOS.

ubuntu installer does not detect LUNs available in such a way. i think its "Detect disks/login to iSCSI target" functionality supposes starting of software iSCSI initiator in the OS instead of detecting and working with LUNs connected via Emulex CNA adapter.

at the same time system successfully detects the presence of Emulex CNA adapter. also in the OS i see loaded components be2iscsi/*.

SLES 11 SP3 and CentOS 6.4 successfully detect such hardware configuration. they are installed, run and work (i have corresponding dmesg and lsmod).

are there any chances that this hardware configuration will also work in ubuntu in the nearest future?

Regards, Andy

[Andy Igoshin (andy-igoshin)]

i tried both 13.04 and 13.10. in what follows let's discuss 13.10.

[Andy Igoshin (andy-igoshin)]

upon closer investigation i discovered some details.

in the initrd of the installer and in the
scsi-modules-3.11.0-11-generic-di_3.11.0-11.17_amd64.udeb
some necessary drivers are absent:

/lib/modules/3.11.0-11-generic/kernel/drivers/firmware/iscsi_ibft.ko
/lib/modules/3.11.0-11-generic/kernel/drivers/net/ethernet/broadcom/cnic.ko
/lib/modules/3.11.0-11-generic/kernel/drivers/scsi/bnx2i/bnx2i.ko
/lib/modules/3.11.0-11-generic/kernel/drivers/uio/uio.ko

/etc/iscsi/initiatorname.iscsi is detected incorrectly. it should be taken from Emulex CNA adapter.
please note that a blade may have some adapters so some different initiator names may exist simultaneously.
these names in a form of "iface.initiatorname = ..." may be written in files describing corresponding interfaces in /etc/iscsi/ifaces/*

i managed to connect iscsi disk manually and then to install OS on it.

i went to "Execute a shell", copied drivers mentioned above, run commands

# modprobe iscsi_tcp
# modprobe bnx2i

# iscsiadm -m iface
# iscsiadm -m fw -l

after that i returned to installer. OS was installed successfully.

but open-iscsi scripts which are used when creating initrd and also during OS start are not intended to work with
Emulex CNA adapter. in these scripts some additional modules should be loaded and some iscsiadm commands
should be run.

also please note that above iscsi LUNs connected in such a way multipath-tools and multipath-tools-boot should work.

[Andy Igoshin (andy-igoshin)]

note

drivers mentioned above are not actually necessary. everything we need is already in the installer (be2net, be2iscsi).
these commands just need to be run in the installer's shell:

# ln -s /sbin/iscsid /usr/sbin
# iscsiadm -m fw -l

to be continued...

[Andy Igoshin (andy-igoshin)]

ok, let's name it with correct words.

irrespective of hardware ubuntu server 13.10 as of 14.10.2013 does not support hardware accelerated iSCSI. there is simply no corresponding functionality. ubuntu supposes that iSCSI should work exclusively over NIC.

PREREQ

iSCSI initiator for the disk where OS is going to be installed to should be configured in Emulex CNA BIOS.

COMPLETE WORKAROUND

1. during installation when installer fails to detect our disks we need to go to installer shell and run:

# ln -s /sbin/iscsid /usr/sbin
# iscsiadm -m fw -l

after these commands iSCSU LUN should be connected. iscsiadm's output should confirm it. the result is available in dmesg.

now we leave installer shell and go back to the installer.

2. in the installer we select Detect Disc, disk is detected. installation now may be proceeded.

3. installation is finished. installer asks us to reboot. we should select other option and go to installer shell again. in the shell we should run:

# chroot /target
# export HOME=/root
# cd

if we wish to boot the system we installed we need to change files
/usr/share/initramfs-tools/hooks/iscsi
/usr/share/initramfs-tools/scripts/local-top/iscsi
(corresponding patches are attached: iscsi-hooks.diff, iscsi-scripts.diff)

after that initrd should be updated (see Bug #1240502) :

# INITRAMFS_EVENT_DRIVEN=1 update-initramfs -u

now we leave installer shell and go back to the installer. we finish the installation.

4. when installed OS is loaded we should forbid updates of the package open-iscsi if we wish not to lose our changes and finally get unbootable OS. if it were changes in mountall file of the package (Bug #1240502) we also should forbid its updates. or *always* run INITRAMFS_EVENT_DRIVEN=1 update-initramfs -u manually when initrd was updated.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "iscsi-hooks.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Andy Igoshin (andy-igoshin)]

mountall team refused my proposal (Bug #1240502) so i added required functionality into iscsi-hooks.diff patch.

[Andy Igoshin (andy-igoshin)]

MULTIPATH

multipath works immediately and as expected. there are no any claims
to multipath itself (except Bug #1240047, Bug #1240049).

problems are begun if you configure more than one initiator for one
physical Emulex CNA interface. obviously this problem relates not only
to multipath but in general to large quantity of initiators configured
in Emulex CNA BIOS.

the cause of this problem is open-iscsi.
'iscsiadm -m fw' will show only some initiators configured in
Emulex CNA BIOS - only one for each physical interface.
'iscsiadm -m discovery -t fw' will return the same implicitly.

further it is more upsetting.

'iscsiadm -m discovery -t fw' or 'iscsiadm -m discovery -t st ...'
(each one separately) can't build working base of nodes.
it gets confused in multipath and generates wild mishmash of
initiators, targets and paths. please note that some of such paths are
incorrect and do not lead to corresponding targets.
so it is impossible to fully automate the definition of nodes, instead
it is required to edit /etc/iscsi/nodes and /etc/iscsi/send_targets
manually.