[860730] Host experiencing constant session termination and recreation while using jumbo frames

[Clayton Skaggs]

We're seeing constant iSCSI session terminations and subsequent recreations on Ubuntu 16.04 hosts running IO to an E-Series Array using Jumbo Frames. We've pulled trace of the issue to determine a root cause and discovered that the host appears to be overrunning the Max Command Sequence Number set by the Target. The target is responds to this be killing that session and recreating a new one in it's place as an error recovery mechanism, so the crux of the issue is why the host seems to be falling outside of iSCSI compliance and overrunning the Max Command Sequence Number. This is the reason I went ahead and filed the bug against open-iscsi.

Attached are actually two separate traces from two separate recreate, one a Finisar Analyzer Trace, the other a typical TCP dump view-able through wire-shark. I've also attached the Apport data for open-iscsi in case that could be of any use.

[Clayton Skaggs (clayton-skaggs)]

We're still consistently seeing this issue across any host running iSCSI to an E-Series target. We can not release iSCSI support for NetApp E-Series till this issue is addressed.

[Clayton Skaggs (clayton-skaggs)]

I wanted to bump this defect again as it doesn't appear anyone has looked at it yet. This issue is still holding up iSCSI support for Ubuntu 16.04 for NetApp E-Series arrays.

[Clayton Skaggs (clayton-skaggs)]

I wanted to bump this defect again as it doesn't appear anyone has looked at it yet. This issue is still holding up iSCSI support for Ubuntu 16.04 for NetApp E-Series arrays.

[Matt Schulte (gimpyestrada)]

We're still hoping for a resolution on this one.

[Nish Aravamudan (nacc)]

Hello,

Given that 17.04 is also on the same upstream base, I dont't think it will be fixed there, so I'm not sure it's worth testing that version.

But Debian has 2.0.874 in unstable. Would you be willing to test a version built off that? If it is fixed upstream, maybe we can figure out what fixed it. Or if not we will need to look elsewhere in the stack.

[Matt Schulte (gimpyestrada)]

Sure, we can give Debian a shot.

[Nish Aravamudan (nacc)]

Hi Matt,

I'll build it for you in a PPA.

-Nish

On Fri, Mar 31, 2017 at 10:45 AM, Matt Schulte
<email address hidden> wrote:
> Sure, we can give Debian a shot.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1615694
>
> Title:
> [860730] Host experiencing constant session termination and recreation
> while using jumbo frames
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1615694/+subscriptions

[Matt Schulte (gimpyestrada)]

Wait I'm confused, build what in a PPA? Didn't you just want us to run Debian and test what the version of open-iscsi it comes with?

[Nish Aravamudan (nacc)]

On Fri, Mar 31, 2017 at 1:17 PM, Matt Schulte
<email address hidden> wrote:
> Wait I'm confused, build what in a PPA? Didn't you just want us to run
> Debian and test what the version of open-iscsi it comes with?

Oh, you're a power user :) I assumed it would be easier for me to
build an Ubuntu version in a PPA.

If you can test Debian directly, that would be great!

[Clayton Skaggs (clayton-skaggs)]

I've setup and attempted the reproduction of this defect on debian using the requested 2.0.874 open-iscsi package. The attempts to reproduce have been unsuccessful.It seems like this build may have a fix for our issue.

[Clayton Skaggs (clayton-skaggs)]

Scratch that. I've now reproduced it on debian with the 2.0.874 package. I'd had the time to setup an ubuntu 16.04 host just to sanity check and reproduced it there just to be sure I had the debian configured correctly. By the time I'd returned to it I was seeing the same logouts as ubuntu. Didn't happen as quickly as I'd anticipated on debian, but it's still logging out and logging back in a few times an hour. Let me know how you'd like to proceed from here.

[Nish Aravamudan (nacc)]

Hi Clayton, sorry for the long delay on my part!

I think at this point, we need to ask upstream open-iscsi (https://github.com/open-iscsi/open-iscsi/issues) about possible bugs in their code.

They probably will ask you be running directly from github's sources (rather than the package) -- are you able to build from source locally? If we can get a fix from upstream narrowed down, I can help you get it backported for SRU.

Thanks,
Nish

[Clayton Skaggs (clayton-skaggs)]

Can't say I know how to build from source like that, no. Any chance you can instruct me on how to do so?

[Christian Ehrhardt  (paelzer)]

Hi Clayton,
I'm cleaning up bugs that work was going on but were forgotten for whatever reason.
Sorry for the delay (again I should say).

To build and use from source you can:
# Build from git
$ git clone https://github.com/open-iscsi/open-iscsi.git
$ sudo apt build-dep open-iscsi
$ sudo apt-get install libssl-dev
$ cd open-iscsi
# Now this step depends on compiler, libs and more - e.g. I had to comment a few lines in include/iscsi_proto.h whicih were now in the linux-headers before being able to build correctly
$ make
$ make install
# Remove the packages version, but backup config
$ cp -a /etc/iscsi ~/iscsi-etc-backup
$ apt remove --purge open-iscsi
# Install what you just built and make it usable
$ systemctl daemon-reload
$ systemctl start open-iscsi
$ systemctl status open-iscsi

As usual you have the config in "/etc/iscsi/" and can try to experiment with that.
Use your backup at ~/iscsi-etc-backup to configure it as similar as possible to the former case.

[Christian Ehrhardt  (paelzer)]

Any updates on testing from upstream (or from newer versions in newer Ubuntu releases) fixing the issue?

[Clayton Skaggs (clayton-skaggs)]

Hi Christian, sorry for the delay. We did make an attempt to go back and build from down stream at one point but without much success. We did go ahead and pick up and qualify Ubuntu 18.04 however and are happy to report that the issue appears to be fixed there with the inbox iscsi package 2.0.874-5ubuntu2. I'll attempt to rerun 16.04 with this build here in the near future to see if that doesn't do the trick.

[Launchpad Janitor (janitor)]

[Expired for open-iscsi (Ubuntu) because there has been no activity for 60 days.]