Targets are not consistently shown with large numbers of targets

[Jordan Callicoat]

SRU JUSTIFICATION

[Impact]

 * tgtadm returns inconsistents results. Some targets are not
   listed during one run, but are shown by a run immediately after.

[Test Case]

 * With large number of targets :

# ./tgtadm -L iscsi --mode target --op show | fgrep -c Target
564

# dpkg -l | fgrep tgt
ii tgt 1:1.0.43-0ubuntu4 amd64 Linux SCSI target user-space tools

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

----
Buggy 1.0.43 version:

# for i in {1..100} ; do diff -u <(/usr/sbin/tgtadm -L iscsi --mode target --op show | fgrep Target) <(/usr/sbin/tgtadm -L iscsi --mode target --op show | fgrep Target) ; sleep 1 ; done
--- /dev/fd/63 2015-09-22 23:05:38.648448562 +0000
+++ /dev/fd/62 2015-09-22 23:05:38.648448562 +0000
@@ -560,3 +560,5 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
+Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45
+Target 581: iqn.2010-10.org.openstack:volume-ddcebcc4-eb13-459e-be1d-2068ee03a4c6
--- /dev/fd/63 2015-09-22 23:05:40.720506264 +0000
+++ /dev/fd/62 2015-09-22 23:05:40.720506264 +0000
@@ -560,3 +560,4 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
+Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45
--- /dev/fd/63 2015-09-22 23:05:41.752535004 +0000
+++ /dev/fd/62 2015-09-22 23:05:41.752535004 +0000
@@ -560,3 +560,5 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
+Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45
+Target 581: iqn.2010-10.org.openstack:volume-ddcebcc4-eb13-459e-be1d-2068ee03a4c6
--- /dev/fd/63 2015-09-22 23:05:43.828592816 +0000
+++ /dev/fd/62 2015-09-22 23:05:43.828592816 +0000
@@ -560,4 +560,3 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
-Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45

[Regression Potential]

 * None expected, the patch comes from tgt upstream repo and it is already implemented in Ubuntu Wily and Debian.

[Other Info]

 * The fix has been tested by the community and the fix resolved the issue. For more details see Comment #4

[Original Description]

With a large number of targets tgtadm --mode target --op show returns inconsistent results. Some targets are not listed during one run, but are shown by a run immediately after.

I grabbed the binary from the wily package and I am getting consistent results, so somewhere between 1.0.43 and 1.0.57 upstream seems to have patched the bug.

Can we get 1.0.57 (or whatever version has the fix) backported to trusty?

----
# ./tgtadm -L iscsi --mode target --op show | fgrep -c Target
564

# dpkg -l | fgrep tgt
ii tgt 1:1.0.43-0ubuntu4 amd64 Linux SCSI target user-space tools

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

----
Buggy 1.0.43 version:

# for i in {1..100} ; do diff -u <(/usr/sbin/tgtadm -L iscsi --mode target --op show | fgrep Target) <(/usr/sbin/tgtadm -L iscsi --mode target --op show | fgrep Target) ; sleep 1 ; done
--- /dev/fd/63 2015-09-22 23:05:38.648448562 +0000
+++ /dev/fd/62 2015-09-22 23:05:38.648448562 +0000
@@ -560,3 +560,5 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
+Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45
+Target 581: iqn.2010-10.org.openstack:volume-ddcebcc4-eb13-459e-be1d-2068ee03a4c6
--- /dev/fd/63 2015-09-22 23:05:40.720506264 +0000
+++ /dev/fd/62 2015-09-22 23:05:40.720506264 +0000
@@ -560,3 +560,4 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
+Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45
--- /dev/fd/63 2015-09-22 23:05:41.752535004 +0000
+++ /dev/fd/62 2015-09-22 23:05:41.752535004 +0000
@@ -560,3 +560,5 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
+Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45
+Target 581: iqn.2010-10.org.openstack:volume-ddcebcc4-eb13-459e-be1d-2068ee03a4c6
--- /dev/fd/63 2015-09-22 23:05:43.828592816 +0000
+++ /dev/fd/62 2015-09-22 23:05:43.828592816 +0000
@@ -560,4 +560,3 @@
 Target 574: iqn.2010-10.org.openstack:volume-7adaa3bc-fafc-4421-baf7-b25102de1922
 Target 578: iqn.2010-10.org.openstack:volume-892c6840-ca04-4474-9976-9eaa8b54c4d2
 Target 579: iqn.2010-10.org.openstack:volume-d1d8062f-be47-4f33-b5b9-1befdfb09f65
-Target 580: iqn.2010-10.org.openstack:volume-32a9e401-cf44-4f8f-ba93-fd43ff758e45

----
Working 1.0.57 version:

# for i in {1..100} ; do diff -u <(./tgtadm -L iscsi --mode target --op show | fgrep Target) <(./tgtadm -L iscsi --mode target --op show | fgrep Target) ; sleep 1 ; done
#

[Jordan Callicoat (jcallicoat)]

For reference there is also an old RHEL bug that may related https://bugzilla.redhat.com/show_bug.cgi?id=865960

[Billy Olsen (billy-olsen)]

From a little digging, it appears that this was fixed upstream in commit https://github.com/fujita/tgt/commit/6c4ffa09dc7681226c3cbe21adde5ff614aecaa2, which unfortunately wasn't included until version 1.0.44.

[Eric Desrochers (slashd)]

I'm providing a hotfix[1] for Trusty (14.04) on a ppa[2], based on the upstream commit : 6c4ffa0[3], for the community to test and provide feedbacks before starting the SRU process.

This hotfix is only for testing purpose.

[1] Version : 1:1.0.43-0ubuntu4.1~14.04.1+20150923lp1498697~1
[2] https://launchpad.net/~eric-desrochers-z/+archive/ubuntu/tgt1498697
[2] 6c4ffa0 tgtadm: fix a bug to read the response buffer

[Charles Farquhar (cfarquhar)]

Thanks Eric!

A quick test by @jcallicoat shows that the package seems to have resolved the issue.

root@XXXXXX-cinder01_cinder_volumes_container-45d8c323:/tmp# for i in {1..100} ; do diff -u <(/usr/sbin/tgtadm -L iscsi --mode target --op show | fgrep Target) <(/usr/sbin/tgtadm -L iscsi --mode target --op show | fgrep Target) ; sleep 1 ; done
--- /dev/fd/63 2015-09-24 01:35:14.491460650 +0000
+++ /dev/fd/62 2015-09-24 01:35:14.491460650 +0000
@@ -578,5 +578,3 @@
Target 607: iqn.2010-10.org.openstack:volume-c04e678e-0f5c-40b1-8f13-3c5928cf9177
Target 608: iqn.2010-10.org.openstack:volume-102fc5d2-93e4-4e34-b616-947cd685004c
Target 609: iqn.2010-10.org.openstack:volume-9009a826-dc4e-43e9-9413-896713f94144
-Target 610: iqn.2010-10.org.openstack:volume-c4210e46-aa24-4176-b719-f8c201b44fdf
-Target 611: iqn.2010-10.org.openstack:volume-c3fb534a-ccca-4fbd-93ef-fb7ae6370936
--- /dev/fd/63 2015-09-24 01:35:19.599588811 +0000
+++ /dev/fd/62 2015-09-24 01:35:19.599588811 +0000
@@ -578,3 +578,4 @@
Target 607: iqn.2010-10.org.openstack:volume-c04e678e-0f5c-40b1-8f13-3c5928cf9177
Target 608: iqn.2010-10.org.openstack:volume-102fc5d2-93e4-4e34-b616-947cd685004c
Target 609: iqn.2010-10.org.openstack:volume-9009a826-dc4e-43e9-9413-896713f94144
+Target 610: iqn.2010-10.org.openstack:volume-c4210e46-aa24-4176-b719-f8c201b44fdf
--- /dev/fd/63 2015-09-24 01:35:29.815845124 +0000
+++ /dev/fd/62 2015-09-24 01:35:29.815845124 +0000
@@ -578,6 +578,3 @@
Target 607: iqn.2010-10.org.openstack:volume-c04e678e-0f5c-40b1-8f13-3c5928cf9177
Target 608: iqn.2010-10.org.openstack:volume-102fc5d2-93e4-4e34-b616-947cd685004c
Target 609: iqn.2010-10.org.openstack:volume-9009a826-dc4e-43e9-9413-896713f94144
-Target 610: iqn.2010-10.org.openstack:volume-c4210e46-aa24-4176-b719-f8c201b44fdf
-Target 611: iqn.2010-10.org.openstack:volume-c3fb534a-ccca-4fbd-93ef-fb7ae6370936
-Target 612: iqn.2010-10.org.openstack:volume-85ea7e62-5317-44f5-8a27-114299924a74
--- /dev/fd/63 2015-09-24 01:35:33.899947586 +0000
+++ /dev/fd/62 2015-09-24 01:35:33.899947586 +0000
@@ -578,5 +578,3 @@
Target 607: iqn.2010-10.org.openstack:volume-c04e678e-0f5c-40b1-8f13-3c5928cf9177
Target 608: iqn.2010-10.org.openstack:volume-102fc5d2-93e4-4e34-b616-947cd685004c
Target 609: iqn.2010-10.org.openstack:volume-9009a826-dc4e-43e9-9413-896713f94144
-Target 610: iqn.2010-10.org.openstack:volume-c4210e46-aa24-4176-b719-f8c201b44fdf
-Target 611: iqn.2010-10.org.openstack:volume-c3fb534a-ccca-4fbd-93ef-fb7ae6370936
^C

root@XXXXXX-cinder01_cinder_volumes_container-45d8c323:/tmp# wget https://launchpad.net/~eric-desrochers-z/+archive/ubuntu/tgt1498697/+files/tgt_1.0.43-0ubuntu4.1%7E14.04.1%2B20150923lp1498697%7E1_amd64.deb
--2015-09-24 01:36:22-- https://launchpad.net/~eric-desrochers-z/+archive/ubuntu/tgt1498697/+files/tgt_1.0.43-0ubuntu4.1%7E14.04.1%2B20150923lp1498697%7E1_amd64.deb
Resolving launchpad.net (launchpad.net)... 91.189.89.222, 91.189.89.223
Connecting to launchpad.net (launchpad.net)|91.189.89.222|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://launchpadlibrarian.net/218674274/tgt_1.0.43-0ubuntu4.1%7E14.04.1%2B20150923lp1498697%7E1_amd64.deb [following]
--...

[Eric Desrochers (slashd)]

debdiff for Trusty

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debdiff for Trusty" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Eric Desrochers (slashd)]

debdiff for Vivid

[Eric Desrochers (slashd)]

debdiff for Precise

[Eric Desrochers (slashd)]

2nd debdiff for Vivid (Fixing the unmet depencies of "libibverbs-dev")

[Chris J Arges (arges)]

Sponsored for T/U/V.

[Brian Murray (brian-murray)]

Why was the version of libibverbs-dev changed in the Vivid upload?

[Eric Desrochers (slashd)]

Hi Brian,

I got a discussion with arges yesterday about it.

vivid version of tgt didn't build with this error :
sbuild-build-depends-tgt-dummy : Depends: libibverbs-dev (>= 1.1.8-1ubuntu1.1) but 1.1.8-1ubuntu1 is to be installed

I think it has something to do with this LP patch :

LP: https://bugs.launchpad.net/bugs/1409904
PPA: https://launchpad.net/~inaddy/+archive/ubuntu/lp1409904/

This is why the version of libibverbs-dev changed in the Vivid upload.

[Brian Murray (brian-murray)]

Were did the vivid version not build? I was able to build tgt version 1.0.43-0ubuntu4.1 in a vivid chroot and see that same version of tgt built in Launchpad successfully.

https://launchpad.net/ubuntu/+source/tgt/1:1.0.43-0ubuntu4.1

Does the environment you were building tgt in have -updates enabled?

[Eric Desrochers (slashd)]

As per our irc conversation, I'm removing the "2nd debdiff for Vivid" and arges will re-upload the fix with the original patch I posted for Vivid.

Sorry for the confusion, and thanks Brian for you feedbacks on this.

[Brian Murray (brian-murray)]

Hello Jordan, or anyone else affected,

Accepted tgt into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tgt/1:1.0.43-0ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Jordan, or anyone else affected,

Accepted tgt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tgt/1:1.0.43-0ubuntu4.1~14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

The precise package version needs fixing as that package version existed in Quantal.

"The source tgt - 1:1.0.17-1ubuntu3 is already accepted in ubuntu/quantal and you cannot upload the same version within the same distribution. You have to modify the source version and re-upload."

[Eric Desrochers (slashd)]

Source version has been modified for version "1:1.0.17-1ubuntu2.1".

Thanks !

[Jordan Callicoat (jcallicoat)]

Tested version 1:1.0.43-0ubuntu4.1~14.04.2 on trusty, looks good to me!

[Robie Basak (racb)]

Is this fixed in Wily? Status says New?

[Robie Basak (racb)]

Ah, "None expected, the patch comes from tgt upstream repo and it is already implemented in Ubuntu Wily and Debian." so setting Fix Released for Wily.

[Launchpad Janitor (janitor)]

This bug was fixed in the package tgt - 1:1.0.43-0ubuntu4.1~14.04.2

---------------
tgt (1:1.0.43-0ubuntu4.1~14.04.2) trusty; urgency=high

  * debian/patches/fix-read-response-buffer.patch : Fix bug to read
    the response buffer (LP: #1498697)

 -- Eric Desrochers <email address hidden> Wed, 30 Sep 2015 11:41:52 -0500

[Chris J Arges (arges)]

The verification of the Stable Release Update for tgt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package tgt - 1:1.0.43-0ubuntu4.2

---------------
tgt (1:1.0.43-0ubuntu4.2) vivid; urgency=medium

  * debian/patches/fix-read-response-buffer.patch : Fix bug to read
    the response buffer (LP: #1498697)

 -- Eric Desrochers <email address hidden> Wed, 30 Sep 2015 11:55:09 -0500

[Brian Murray (brian-murray)]

Hello Jordan, or anyone else affected,

Accepted tgt into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tgt/1:1.0.17-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package tgt - 1:1.0.17-1ubuntu2.1

---------------
tgt (1:1.0.17-1ubuntu2.1) precise; urgency=medium

  * debian/patches/fix-read-response-buffer.patch : Fix bug to read
    the response buffer (LP: #1498697)

 -- Eric Desrochers <email address hidden> Wed, 28 Oct 2015 10:47:23 -0500