[Lenovo Ubuntu 24.04 Bug] Can not detect iscsi disk storage and kernel show " detected conn error (1020)"

[xiaochun Lee]

Description:
When installing Ubuntu 24.04 in remote iscsi storage disk, the installer can not detect the remote iscsi storage. After install Ubuntu 24.04 to a local disk and boot up to OS, the OS can not detect the remote iscsi disk either.

Reproduce Steps:
  1. Install Ubunut 24.04
  2. Boot into OS
  3. perform command lsblk check remote disk, there isn't one appeared.

Configuration:
System: Lenovo SR655V3
OS:noble-live-server-amd64-0401.iso
BMC Version ：2.30 (Build ID: KAX321V)
UEFI Version:3.10 (Build ID: KAE115K)
CPU:Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz*2 M321R2GA3BB6-CQKMG*8
NIC adapter: Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port PCIe Ethernet Adapte
storage：Lenovo D7000 or Lenovo V7000

 Expected results:
 OS can recognize the remote iscsi disk

 Actual results:
 OS cannot recognize the remote iscsi disk.

Additional info:
  1. Ubuntu 22.04 also have this issue, But RHEL9.4 hasn't.

[xiaochun Lee (xavier-lee)]

 The OS will pop out the "detected conn error (1020)" after run command "systemctl restart iscsid" and command "systemctl restart open-iscsi" I've open debug mode of module "scsi_tcp, libiscsi, scsi_transport_iscsi" and have the journal log and sosreport log for you.

-->>
 Apr 10 07:48:27 test kernel: scsi host18: iSCSI Initiator over TCP/IP
 Apr 10 07:48:28 test kernel: connection2:0: detected conn error (1020)
 Apr 10 07:48:28 test iscsid[3302]: Could not set session2 priority. READ/WRITE throughout and latency could be affected.
 Apr 10 07:48:28 test iscsid[3302]: Login failed to authenticate with target iqn.1986-03.com.ibm:2145.v7000gen2.node2
 Apr 10 07:48:28 test iscsid[3302]: session 2 login rejected: Initiator failed authentication with target
 Apr 10 07:48:28 test iscsid[3302]: Connection2:0 to [target: iqn.1986-03.com.ibm:2145.v7000gen2.node2, portal: 9.0.0.22,3260] through [iface: default] is shutdown

 Apr 10 07:59:34 test iscsiadm[4267]: iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
 Apr 10 07:59:34 test iscsiadm[4267]: iscsiadm: Could not login to [iface: default, target: iqn.1986-03.com.ibm:2145.v7000gen2.node2, portal: 9.0.0.22,3260].
 Apr 10 07:59:34 test iscsiadm[4267]: iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
 Apr 10 07:59:34 test iscsiadm[4267]: iscsiadm: Could not log into all portals
 Apr 10 07:59:34 test iscsiadm[4267]: Logging in to [iface: default, target: iqn.1986-03.com.ibm:2145.v7000gen2.node2, portal: 2000:0501:ffff:0100:0000:0000:0000:0666,3260]
Apr 10 07:59:34 test iscsiadm[4267]: Logging in to [iface: default, target: iqn.1986-03.com.ibm:2145.v7000gen2.node2, portal: 9.0.0.22,3260]
<<--

[xiaochun Lee (xavier-lee)]

Hello, Canonical
Is there any updates for this issue? Appreciate it!

[xiaochun Lee (xavier-lee)]

From our investigation, we found that Ubuntu didn't add iscsi_ibft module automatically, but after I manually modprobe iscsi_ibft, the command has output as follows, but we still couldn't login iscsi disk and got kernel socket connect error.
-->
root@test:~# iscsiadm --mode fw
# BEGIN RECORD 2.1.9
iface.initiatorname = iqn.1986-03.com.example:lql
iface.transport_name = tcp
iface.hwaddress = 0c:42:a1:ca:e7:e0
iface.bootproto = DHCP
iface.ipaddress = 9.0.1.108
iface.prefix_len = 23
iface.subnet_mask = 255.255.254.0
iface.gateway = 9.0.0.2
iface.primary_dns = 9.0.0.1
iface.vlan_id = 0
iface.net_ifacename = ens4f0np0
node.name = iqn.1986-03.com.ibm:2145.v7000gen2.node2
node.conn[0].address = 9.0.0.22
node.conn[0].port = 3260
node.boot_lun = 00000000
# END RECORD
root@test:~#
<<--

[xiaochun Lee (xavier-lee)]

From continue investigate on it, we found the root cause is that installer in Ubuntu can not sync the iscsi IQN name (iscsi qualified name) defined in UEFI to OS /etc/iscsi/initiatorname.iscsi via iscsi iBFT (iscsi boot firmware table), even we add dracut parameter "rd.driver.pre=iscsi_ibft rd.iscsi.firmware=1 rd.iscsi.ibft=1 rd.iscsi.initiator=iqn.1986-03.com.example:lql rd.iscsi.param=node.startup=automatic"

Ubuntu can detect the remote iscsi disk, If we overwrite the default line in /etc/iscsi/initiaorname.iscsi to "InitiatorName=iqn.1986-03.com.example:lql" manually and restart iscsid and open-iscsi service.

Note: the string "iqn.1986-03.com.example:lql" is the name defined in UEFI for iscsi target lun.

But after install OS to remote iscsi disk, we can not boot up from the installed OS, because initrd doesn't include iscsi drivers (libiscsi, scsi_transport_iscsi, iscsi_tcp, libiscsi_tcp) and services open-iscsi and iscsid and their needed utilities, like iscsiadm, iscsid.

So I was just wondering of Canonical experts if Ubuntu support iscsi installation or not?

[Kleber Sacilotto de Souza (kleber-souza)]

This issue is definitely not a regression in 24.04 as it also affects 22.04 as reported.

For now this doesn't seem to be a kernel issue. The module "iscsi_ibft.ko" is shipped by the linux-modules-<ver> package, therefore if it's not loaded and configured during installation it seems to be a user-space issue.

I'll keep it as "Triaged" though on the kernel side until we have more information from the investigation.