[Ubuntu 18.04]Unable to create NetworkPortal in configFS using Targetcli tool

[bugproxy]

== Comment: #0 - Prudhvi Miryala <email address hidden> - 2018-03-21 09:08:25 ==
---Problem Description---
[Ubuntu 18.04]Unable to create NetworkPortal in configFS using Targetcli tool

---uname output---
Linux ltc-test-ci1 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = 8247-21L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1) create image
dd if=/dev/zero of=/var/lib/libvirt/images/avocado_6fE62R/emulated-image count=1024 bs=1024K
2) open the targetcli
targetcli
 and run following commands on targetcli tool
3) cd backstores/

4) cd fileio

5) create back end device.
create testlun1 /var/lib/libvirt/images/avocado_6fE62R/emulated-image

6) cd iscsi

7) generate iqn for server
create

8) cd iqn.2003-01.org.linux-iscsi.ltc-test-ci1.ppc64le:sn.7dc2e829a6b3

9) cd tpg1/luns
create LUN
create /backstores/fileio/testlun1
10) cd ../portal

11) create 0.0.0.0

expected output:
------------------------------
network portal should create successfully.

actual output:
--------------------
Could not create NetworkPortal in configFS

Contact Information = <email address hidden>

Userspace tool common name: targetcli

The userspace tool has the following bit modes: 64bit

Userspace rpm: targetcli-fb

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach ltrace and strace of userspace application.

== Comment: #5 - Douglas Miller <email address hidden> - 2018-04-03 12:17:24 ==
A quick web search shows that many people have run into a problem like this, because there was a default port created automatically and that prevents an explicit create of ports. I am not familiar with this tool, but the solution for those cases was to delete the 0.0.0.0:3260 port.

== Comment: #6 - Prudhvi Miryala <email address hidden> - 2018-04-04 04:38:24 ==
Hi Team,

Working fine on latest kernel at following levels:

Linux ltc-boston114 4.15.0-14-generic #15-Ubuntu SMP Mon Apr 2 19:47:43 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux.

Still we are facing issue on following levels:

Linux ltc-test-ci1 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

So i am closing the Bug

== Comment: #7 - Prudhvi Miryala <email address hidden> - 2018-04-04 04:49:34 ==

== Comment: #8 - Prudhvi Miryala <email address hidden> - 2018-04-09 08:40:54 ==
i am reopen the issue because i am seeing the same issue in latest build(4.15.0-15-generic)
---> first i am seeing issue on kernel build 4.15.0-13-generic
**targetcli package version: 2.1.43-1

----> later i updated the kernel build to 4.15.0-14-generic then issue is not reproduce on 4.15.0-14-generic but able to reproduce on 4.15.0-13-generic.
**targetcli package version: 2.1.43-1

----> later i updated the kernel build to 4.15.0-15-generic then i am able to reproduce the issue on all three kernel builds(4.15.0-15-generic,4.15.0-14-generic and 4.15.0-13-generic)
**targetcli package version: 2.1.43-1

PS: steps to reproduce the issue available above, if need more details about recreation steps please let me know

== Comment: #12 - Prudhvi Miryala <email address hidden> - 2018-04-09 10:10:51 ==
using default port i am unable to create portal and i am seeing following error

/tpg1/portals> create 0.0.0.0 3260
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Could not create NetworkPortal in configFS
path: /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.ltc-boston17.ppc64le:sn.8cb45748eb75/tpgt_1/np/0.0.0.0:3260
details:[Errno 22] Invalid argument: '/sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.ltc-boston17.ppc64le:sn.8cb45748eb75/tpgt_1/np/0.0.0.0:3260'
----------------------------------------------------------------------------------------------------------------
i tried with different port(not default port) then i am able to create network portal.

== Comment: #13 - Prudhvi Miryala <email address hidden> - 2018-04-09 10:12:29 ==
(In reply to comment #12)
> using default port i am unable to create portal and i am seeing following
> error
>
> /tpg1/portals> create 0.0.0.0 3260
> Using default IP port 3260
> Binding to INADDR_ANY (0.0.0.0)
> Could not create NetworkPortal in configFS
> path:
> /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.ltc-boston17.
> ppc64le:sn.8cb45748eb75/tpgt_1/np/0.0.0.0:3260
> details:[Errno 22] Invalid argument:
> '/sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.ltc-boston17.
> ppc64le:sn.8cb45748eb75/tpgt_1/np/0.0.0.0:3260'
> -----------------------------------------------------------------------------
> -----------------------------------
> i tried with different port(not default port) then i am able to create
> network portal.

Default port num: 3260
i tried with port num : 3261(not default port)

== Comment: #18 - MAMATHA INAMDAR <email address hidden> - 2018-04-10 02:47:01 ==
/iscsi> ls
o- iscsi .............................................................................................................. [Targets: 2]
  o- iqn.2003-01.org.linux-iscsi.ltc-boston17.ppc64le:sn.8cb45748eb75 .................................................... [TPGs: 1]
  | o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
  | o- acls ............................................................................................................ [ACLs: 0]
  | o- luns ............................................................................................................ [LUNs: 0]
  | o- portals ...................................................................................................... [Portals: 5]
  | o- 0.0.0.0:2260 ....................................................................................................... [OK]
  | o- 0.0.0.0:3261 ....................................................................................................... [OK]
  | o- 0.0.0.0:3262 ....................................................................................................... [OK]
  | o- 0.0.0.0:3660 ....................................................................................................... [OK]
  | o- 0.0.0.0:3661 ....................................................................................................... [OK]
  o- iqn.2003-01.org.linux-iscsi.ltc-boston17.ppc64le:sn.adf4ca056925 .................................................... [TPGs: 1]
    o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
      o- acls ............................................................................................................ [ACLs: 0]
      o- luns ............................................................................................................ [LUNs: 0]
      o- portals ...................................................................................................... [Portals: 0]

If we try to delete and create network portal for any of the above ports it will work fine
Ex: 3660 as below

/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 ip_port=3660
Deleted network portal 0.0.0.0:3660

/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 4]
  o- 0.0.0.0:2260 ............................................................................................................. [OK]
  o- 0.0.0.0:3261 ............................................................................................................. [OK]
  o- 0.0.0.0:3262 ............................................................................................................. [OK]
  o- 0.0.0.0:3661 ............................................................................................................. [OK]

/iscsi/iqn.20.../tpg1/portals> create 0.0.0.0 ip_port=3660
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3660.

/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 5]
  o- 0.0.0.0:2260 ............................................................................................................. [OK]
  o- 0.0.0.0:3261 ............................................................................................................. [OK]
  o- 0.0.0.0:3262 ............................................................................................................. [OK]
  o- 0.0.0.0:3660 ............................................................................................................. [OK]
  o- 0.0.0.0:3661 ............................................................................................................. [OK]

But Prudhvi is trying to create network portal with default port 3260 which is not working,
which shows invalid argument

/iscsi/iqn.20.../tpg1/portals> create 0.0.0.0 ip_port=3260
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Could not create NetworkPortal in configFS
 path: /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.ltc-boston17.ppc64le:sn.8cb45748eb75/tpgt_1/np/0.0.0.0:3260
 details:[Errno 22] Invalid argument: '/sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.ltc-boston17.ppc64le:sn.8cb45748eb75/tpgt_1/np/0.0.0.0:3260'

I am not sure whether it's compulsory to create network portal with default port 3260?

== Comment: #21 - Murilo Fossa Vicentini <email address hidden> - 2018-04-10 09:38:13 ==
Since I don't work in this area, I may be speaking nonsense here, so please take the following with a grain of salt.

Is there any chance this is done transparently? At least looking at netstat I see that the default port (3260) is in use:

root@ltc-boston17:~# netstat -a -n | grep "3260\|Address"
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN
tcp6 0 0 :::3260 :::* LISTEN

Also, just throwing this out here, since I saw you have a qla2xxx adapter in your system, I saw this disclaimer in the man pages for targetcli:

CONFIGURING FIBRE CHANNEL (QLA2XXX)
       Operation as a target requires that /sys/module/qla2xxx/parameters/qlini_mode report "disabled". This may require passing the qlini_mode=disabled parameter to the qla2xxx module when it loads.

And here is the description for this

# modinfo qla2xxx | grep qlini_mode:
parm: qlini_mode:Determines when initiator mode will be enabled. Possible values: "exclusive" - initiator mode will be enabled on load, disabled on enabling target mode and then on disabling target mode enabled back; "disabled" - initiator mode will never be enabled; "dual" - Initiator Modes will be enabled. Target Mode can be activated when ready "enabled" (default) - initiator mode will always stay enabled. (charp)

Could this somehow explain the behavior seen here?

== Comment: #23 - SANTWANA SAMANTRAY <email address hidden> - 2018-07-27 00:36:35 ==
Hi All,

I validated this issue on Ubuntu18.04 (k.v- 4.15.0-29-generic) and its still reproducible.
targetcli package version: 2.1.43-1

/iscsi> ls
o- iscsi .............................................................................................................. [Targets: 3]
  o- iqn.2003-01.org.linux-iscsi.ltc-zz2.ppc64le:sn.7a0db87229ad ......................................................... [TPGs: 1]
  | o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
  | o- acls ............................................................................................................ [ACLs: 0]
  | o- luns ............................................................................................................ [LUNs: 0]
  | o- portals ...................................................................................................... [Portals: 0]
  o- iqn.2018-07.com.virttest:emulated-image-disk.target ................................................................. [TPGs: 1]
  | o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
  | o- acls ............................................................................................................ [ACLs: 0]
  | o- luns ............................................................................................................ [LUNs: 0]
  | o- portals ...................................................................................................... [Portals: 0]
  o- iqn.2018-07.com.virttest:emulated-iscsi.target ...................................................................... [TPGs: 1]
    o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
      o- acls ............................................................................................................ [ACLs: 0]
      o- luns ............................................................................................................ [LUNs: 0]
      o- portals ...................................................................................................... [Portals: 0]

Using non-default port:
/iscsi/iqn.20.../tpg1/portals> create 0.0.0.0 ip_port=3660
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3660.
/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 1]
  o- 0.0.0.0:3660 ............................................................................................................. [OK]

Using default port:
/iscsi/iqn.20.../tpg1/portals> create 0.0.0.0 ip_port=3260
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Could not create NetworkPortal in configFS

The default port(3260) is in use, when we get the above error:
# netstat -a -n | grep "3260\|Address"
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN
tcp6 0 0 :::3260 :::* LISTEN

Thanks,
Santwana

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Manoj Iyer (manjo)]

Looks like description subcomment == Comment: #21 seems to indicate the qla2xxx might have something to do with this bug as well?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-08-13 11:18 EDT-------
I'm afraid the information in this bug is not concise or properly scoped. One internal comment states/clarifies:

"[...] Prudhvi is not able to create network portal with default port which is 3260. [...] But he is able to create network portal with non-default port i.e 3261."

I was wondering if this is simply a case of a target having been already created (by the system?) with the default port, and thus a conflict which prevents creating a new target with the same port. Not sure if there is something else the tool should be doing in this case, even if just a better error message.

I have seen (older) discussions on the web of similar issues, which talk about manually deleting the default target before creating a new one. But I'm not familiar with this tool or the facilities.

[Dimitri John Ledkov (xnox)]

Whilst the bug is set to trianged and assigned to canonical foundations team, I'm not sure if there is anything to do w.r.t. this bug. I'm also not familiar with this stuff, but I guess the next steps is to trace what pre-created things on the default port and figure out if that can be disabled - since doing that by default doesn't result in correct and working configs?!

Marking incomplete, until it is clear as to what is expected next of this bug report.

[Dimitri John Ledkov (xnox)]

I wonder if canonical server team might be of help here.

[Ryan Harper (raharper)]

The comments are unclear what the actual sequence of failure is, but as comment #4 mentions the tool will automatically listen to the default port on *ALL* interfaces (0.0.0.0).

If you want to manually create one using the existing default port, you have to delete the 0.0.0.0 entry before adding a new one on the default port.

Can submitter after rebooting system:

1) confirm the iscsi service is up and running
2) use netstat to see what , if anything is listening on port 3260

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-09-25 22:53 EDT-------
Originator of the bug has left the organization. I tried recreating this issue, but everything works well with 18.04 as well 18.10. We can close this one.

[Larry L (llilly)]

I had this exact same problem. I noticed that tgt had been installed and was running (that's what must have been using 0.0.0.0:3260). So I killed the tgt process and then was able to create the portal with the default 3260 port in targetcli.

[Patrick Hogan (ph0g4n)]

Larry, your post led me to find and fix my "portals 3260" issue. I was searching all day for a fix. Everything in targetcli worked and I can attach my LUNs now. Thank you!

[fast_rizwaan (fast-rizwaan)]

$ sudo apt remove tgt
$ targetcli
> /iscsi/iqn.2003-01.org.linux-iscsi.xxxxx.xxxx:sn.2139958cb0e4/tpg1/portals create

got success
----
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3260.