Bug #1294267 “tgt doesn't export some LUNs after a reload operat...” : Bugs : tgt package : Ubuntu

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2014-03-18:

#1

Dependencies.txt Edit (932 bytes, text/plain; charset="utf-8")
ProcEnviron.txt Edit (110 bytes, text/plain; charset="utf-8")

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2014-03-18:

#2

I think the problem can be related to this fact: the TGT daemon don't add a block storage to a LUN if some process was accessing it.

The following command show another operation in which a similar problem occours:

# All block storage added to the targets
$ tgt-admin --dump | grep -o "backing-store $.*$" | sed -e "s/^backing-store $.*$$/\1/"
/dev/vg_vmware/vmware_4
/dev/vg_vmware/vmware_3
/dev/vg_vmware/vmware_2
/dev/vg_vmware/vmware_1
/dev/vg_vmware/vmware_5
# Now, we'll run a stop/start operation. Some device block (previously LUNs) still opened by some process
# between the `stop` and `start` operations. In our case, the device blocks was opened by the command `blkid`,
# running with the 'root' user.
$ stop tgt; lsof /dev/mapper/vg_vmware-vmware_1 /dev/mapper/vg_vmware-vmware_2 \
$ /dev/mapper/vg_vmware-vmware_3 /dev/mapper/vg_vmware-vmware_4 \
$ /dev/mapper/vg_vmware-vmware_5; start tgt; tgt-admin --dump | grep -o "backing-store $.*$" | \
$ sed -e "s/^backing-store $.*$$/\1/"
tgt stop/waiting
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
blkid 29870 root 3r BLK 252,2 0t212992 1585695 /dev/mapper/../dm-2
blkid 29871 root 3r BLK 252,0 0t143360 1605724 /dev/mapper/../dm-0
blkid 29872 root 3r BLK 252,3 0t180224 1585696 /dev/mapper/../dm-3
blkid 29875 root 3r BLK 252,4 0x1fd70bce200 1585697 /dev/mapper/../dm-4
tgt start/running, process 29878
/dev/vg_vmware/vmware_5
$ ls -lh /dev/mapper/vg_vmware-vmware_1 /dev/mapper/vg_vmware-vmware_2 /dev/mapper/vg_vmware-vmware_3 /dev/mapper/vg_vmware-vmware_4 /dev/mapper/vg_vmware-vmware_5
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_2 -> ../dm-2
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_3 -> ../dm-3
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_4 -> ../dm-4
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_5 -> ../dm-5

In our test, four device blocks (vg_vmware-vmware_1, vg_vmware-vmware_2, vg_vmware-vmware_3 and vg_vmware-vmware_4) were opened by the blkid command, and the `start tgt` command (that runs `/usr/sbin/tgt-admin -e`) only adds the other lun (vg_vmware-vmware_5).

I hope this test helps.

I think the problem can be related to this fact: the TGT daemon don't add a block storage to a LUN if some process was accessing it.

The following command show another operation in which a similar problem occours:

# All block storage added to the targets
$ tgt-admin --dump | grep -o "backing-store $.*$" | sed -e "s/^backing-store $.*$$/\1/"
/dev/vg_vmware/vmware_4
/dev/vg_vmware/vmware_3
/dev/vg_vmware/vmware_2
/dev/vg_vmware/vmware_1
/dev/vg_vmware/vmware_5
# Now, we'll run a stop/start operation. Some device block (previously LUNs) still opened by some process 
# between the `stop` and `start` operations. In our case, the device blocks was opened by the command `blkid`, 
# running with the 'root' user.
$ stop tgt; lsof /dev/mapper/vg_vmware-vmware_1 /dev/mapper/vg_vmware-vmware_2 \
$     /dev/mapper/vg_vmware-vmware_3  /dev/mapper/vg_vmware-vmware_4 \
$     /dev/mapper/vg_vmware-vmware_5; start tgt; tgt-admin --dump | grep -o "backing-store $.*$" | \
$     sed -e "s/^backing-store $.*$$/\1/"
tgt stop/waiting
COMMAND   PID USER   FD   TYPE DEVICE      SIZE/OFF    NODE NAME
blkid   29870 root    3r   BLK  252,2      0t212992 1585695 /dev/mapper/../dm-2
blkid   29871 root    3r   BLK  252,0      0t143360 1605724 /dev/mapper/../dm-0
blkid   29872 root    3r   BLK  252,3      0t180224 1585696 /dev/mapper/../dm-3
blkid   29875 root    3r   BLK  252,4 0x1fd70bce200 1585697 /dev/mapper/../dm-4
tgt start/running, process 29878
/dev/vg_vmware/vmware_5
$ ls -lh /dev/mapper/vg_vmware-vmware_1 /dev/mapper/vg_vmware-vmware_2 /dev/mapper/vg_vmware-vmware_3  /dev/mapper/vg_vmware-vmware_4  /dev/mapper/vg_vmware-vmware_5
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_2 -> ../dm-2
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_3 -> ../dm-3
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_4 -> ../dm-4
lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_5 -> ../dm-5

In our test, four device blocks (vg_vmware-vmware_1, vg_vmware-vmware_2, vg_vmware-vmware_3 and vg_vmware-vmware_4) were opened by the blkid command, and the `start tgt` command (that runs `/usr/sbin/tgt-admin -e`) only adds the other lun (vg_vmware-vmware_5).

I hope this test helps.

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2014-03-18:

#3

Sometimes, the restart operation doesn't works also:

$ tgt-admin --dump

default-driver iscsi

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_4>
backing-store /dev/vg_vmware/vmware_4
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_3>
backing-store /dev/vg_vmware/vmware_3
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_2>
backing-store /dev/vg_vmware/vmware_2
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_1>
backing-store /dev/vg_vmware/vmware_1
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_5>
backing-store /dev/vg_vmware/vmware_5
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

$ restart tgt
tgt start/running, process 30315

$ tgt-admin --dump
default-driver iscsi

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_4>
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_3>
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_2>
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_1>
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

<target iqn.2014-03.br.com.2aliancas.storage2:vmware_iscsi_5>
backing-store /dev/vg_vmware/vmware_5
initiator-address 192.168.130.0/24
initiator-address 127.0.0.1
</target>

In this case, only the /dev/vg_vmware/vmware_5 device block was added as a LUN.

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2014-03-23:

#4

I discovered that, although sending a SIGHUG signal to TGT (using the `reload tgt` command) doesn't works, I can correctly "reload" TGT using the following command:

tgt-admin --update ALL

I know upstart has the option to change the signal send to a deamon. However, if we could change the `reload` behavior to send no signal and exec an script, we could correct the `reload` wrong behavior.

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2014-03-23:

#5

tgt.override Edit (2.1 KiB, text/plain)

In the attachment, I implemented a fix for the `restart` and `stop+start` bug.

It's a poor fix, but it may help.

Unfortunately, I don't found any fix for the reload method, since upstart (in Ubuntu 12.04) always send a SIGHUP signal to the deamon.

Serge Hallyn (serge-hallyn) on 2014-03-27

Changed in tgt (Ubuntu):
importance:	Undecided → High

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2014-03-27:

#6

Thanks for taking the time to report this bug. It doesn't look like it's been addressed upstream at all (no catching of sighup going on in git head).

Changed in tgt (Ubuntu):
status:	New → Confirmed

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2014-03-30:

#7

Download full text (7.6 KiB)

Hi Serge Hallyn,

I run some new tests and, in short, the problem I found is actually three problems (maybe this bug should be broke in three).
I will summarize the problems I found:

- Tgtd breaks when it receives a SIGHUP;
  - When we send a SIGHUP to tgtd (`reload tgt`), the deamon stops (and it logs an fault in syslog), and then upstart respawns the deamon. It is, run `reload tgt` appears to be equivalent to run `kill -9 TGT_ROOT_PID; start tgt;`;
- Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command;
  - When tgtd stops and closes the devices, the system will probably run a `blkid` in this devices. So, when tgtd starts again, some devices may be still opened by the blkid command, and neither tgtd or `tgt-admin -e` will add this device (I'm considering "allow-in-use" is configured to "off" in this moment);
- The `stop tgt` command really stops tgtd even if some LUNs still in use;
  - This can cause data corruption.

Debian wheezy (https://packages.debian.org/source/wheezy/tgt) solves these problem using the initd.sample from upstream.

In the next explanations, I'm using the source code found in: http://packages.ubuntu.com/source/precise/tgt

- Tgtd breaks when it receives a SIGHUP:

From the initd.sample file (tgt-1.0.17/scripts/initd.sample, line 79):

reload()
{
echo "Updating target framework daemon configuration"
# Update configuration for targets. Only targets which
# are not in use will be updated.
tgt-admin --update ALL -c $TGTD_CONFIG &>/dev/null
RETVAL=$?
if [ "$RETVAL" -eq 107 ] ; then
echo "tgtd is not running"
exit 1
fi
}

It is, the `service tgt reload` command will run a `tgt-admin --update ALL` command, and this last command works fine. The `service tgt reload` command DOESN'T send a SIGHUP signal in Debian (and in all Linux distributions that uses the initd.sample file).

I don't see how fix this in Ubuntu using upstart, because in Ubuntu 12.04 Upstart always send a SIGHUP signal. In newer versions of Ubuntu we can use the `reload signal` stanza, but we can't run a script (`tgt-admin --update ALL`) when `reload tgt` was called.

- Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command:

First, let's read the tgt-admin source code (tgt-1.0.17/scripts/tgt-admin, line 1240):

# Check if userspace uses this device
my $lsof_check = check_exe("lsof");
if ($lsof_check ne 1) {
  system("lsof $backing_store &>/dev/null");
  my $exit_value = $? >> 8;
  if ($exit_value eq 0) {
   execute("# Device $backing_store is used (already tgtd target?).");
   execute("# Run 'lsof $backing_store' to see the details.");
   return 0;
  }
}

It is, when `tgt-admin -e` (in the `post-start` stanza) is executed, if some device still opened (in our case, by the `blkid` command), we can add this device as a LUN.

In my attachment tgt.override, a hack was proposed to avoid this problem. However, my tests showed that waiting one second between stops and starts tgtd solves the problem. Actually, 0.3 second waiting solved my problem in all tests (but 1 second is a more secure interval).

From the initd.sample file (tgt-1.0.17/scripts/initd.sample, lin...

Hi Serge Hallyn,

I run some new tests and, in short, the problem I found is actually three problems (maybe this bug should be broke in three). 
I will summarize the problems I found:

- Tgtd breaks when it receives a SIGHUP;
  - When we send a SIGHUP to tgtd (`reload tgt`), the deamon stops (and it logs an fault in syslog), and then upstart respawns the deamon. It is, run `reload tgt` appears to be equivalent to run `kill -9 TGT_ROOT_PID; start tgt;`;
- Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command;
  - When tgtd stops and closes the devices, the system will probably run a `blkid` in this devices. So, when tgtd starts again, some devices may be still opened by the blkid command, and neither tgtd or `tgt-admin -e` will add this device (I'm considering "allow-in-use" is configured to "off" in this moment);
- The `stop tgt` command really stops tgtd even if some LUNs still in use;
  - This can cause data corruption.

Debian wheezy (https://packages.debian.org/source/wheezy/tgt) solves these problem using the initd.sample from upstream.

In the next explanations, I'm using the source code found in: http://packages.ubuntu.com/source/precise/tgt

- Tgtd breaks when it receives a SIGHUP:

From the initd.sample file (tgt-1.0.17/scripts/initd.sample, line 79):

reload()
{
	echo "Updating target framework daemon configuration"
	# Update configuration for targets. Only targets which
	# are not in use will be updated.
	tgt-admin --update ALL -c $TGTD_CONFIG &>/dev/null
	RETVAL=$?
	if [ "$RETVAL" -eq 107 ] ; then
	    echo "tgtd is not running"
	    exit 1
	fi
}

It is, the `service tgt reload` command will run a `tgt-admin --update ALL` command, and this last command works fine. The `service tgt reload` command DOESN'T send a SIGHUP signal in Debian (and in all Linux distributions that uses the initd.sample file).

I don't see how fix this in Ubuntu using upstart, because in Ubuntu 12.04 Upstart always send a SIGHUP signal. In newer versions of Ubuntu we can use the `reload signal` stanza, but we can't run a script (`tgt-admin --update ALL`) when `reload tgt` was called.

- Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command:

First, let's read the tgt-admin source code (tgt-1.0.17/scripts/tgt-admin, line 1240):

# Check if userspace uses this device
	my $lsof_check = check_exe("lsof");
	if ($lsof_check ne 1) {
		system("lsof $backing_store &>/dev/null");
		my $exit_value  = $? >> 8;
		if ($exit_value eq 0) {
			execute("# Device $backing_store is used (already tgtd target?).");
			execute("# Run 'lsof $backing_store' to see the details.");
			return 0;
		}
	}

It is, when `tgt-admin -e` (in the `post-start` stanza) is executed, if some device still opened (in our case, by the `blkid` command), we can add this device as a LUN.

In my attachment tgt.override, a hack was proposed to avoid this problem. However, my tests showed that waiting one second between stops and starts tgtd solves the problem. Actually, 0.3 second waiting solved my problem in all tests (but 1 second is a more secure interval).

From the initd.sample file (tgt-1.0.17/scripts/initd.sample, line 125):

restart)
		TASK=restart
		stop && start
		;;

And from line 19:

# Put tgtd into "offline" state until all the targets are configured.
	# We don't want initiators to (re)connect and fail the connection
	# if it's not ready.
	tgtadm --op update --mode sys --name State -v offline
	# Configure the targets.
	tgt-admin -e -c $TGTD_CONFIG
	# Put tgtd into "ready" state.
	tgtadm --op update --mode sys --name State -v ready

A fix for this problem can be:

# Put tgtd into "offline" state until all the targets are configured.
	# We don't want initiators to (re)connect and fail the connection
	# if it's not ready.
	tgtadm --op update --mode sys --name State -v offline
+	# Wait the system free the backing-stores (a `blkid` command
+	# may still be running)
+	# See: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267
+	[ "$TASK" == "restart" ] && sleep 1
	# Configure the targets.
	tgt-admin -e -c $TGTD_CONFIG
	# Put tgtd into "ready" state.
	tgtadm --op update --mode sys --name State -v ready

- The `stop tgt` command really stops tgtd even if some LUNs still in use:

Let's see the source code again (tgt-1.0.17/scripts/initd.sample, line 35):

# Remove all targets. It only removes targets which are not in use.
	tgt-admin --update ALL -c /dev/null &>/dev/null
	# tgtd will exit if all targets were removed
	tgtadm --op delete --mode system &>/dev/null
	RETVAL=$?
	if [ "$RETVAL" -eq 107 ] ; then
	    echo "tgtd is not running"
	    [ "$TASK" != "restart" ] && exit 1
	elif [ "$RETVAL" -ne 0 ] ; then
	    echo "Some initiators are still connected - could not stop tgtd"
	    exit 2
	fi

It is, in the tgt upstream initd script `service tgt stop` command shouldn't stop the deamon if some target still in use.

However, in the Ubuntu upstart version, according to Upstart cookbook, when we run `stop tgt`, upstart sends a SIGTERM signal and waits the time specified by the `kill timeout` stanza. After this, it sends a SIGKILL signal, forcing the deamon to stop.

See: http://upstart.ubuntu.com/cookbook/#kill-signal, http://upstart.ubuntu.com/cookbook/#kill-timeout

--------------

Solutions (patches):

There are two main kind of solutions: fix upstart, or use the initd.sample script again.

- Tgtd breaks when it receives a SIGHUP:

-- Initd.sample: Already solved
-- Upstart: No way to solve this problem

- Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command:

-- Initd.sample:

Patch applied to initd.sample file (tgt-1.0.17/scripts/initd.sample, line 19):

# Put tgtd into "offline" state until all the targets are configured.
	# We don't want initiators to (re)connect and fail the connection
	# if it's not ready.
	tgtadm --op update --mode sys --name State -v offline
+	# Wait the system free the backing-stores (a `blkid` command
+	# may still be running)
+	# See: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267
+	[ "$TASK" == "restart" ] && sleep 1
	# Configure the targets.
	tgt-admin -e -c $TGTD_CONFIG
	# Put tgtd into "ready" state.
	tgtadm --op update --mode sys --name State -v ready

-- Upstart:

Patch applied to tgt.conf file (debian/upstart, line 11):

-	post-start exec /usr/sbin/tgt-admin -e
+	post-start script
+	# Wait the system free the backing-stores (a `blkid` command
+	# may still be running)
+	# See: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267
+	sleep 1
+	/usr/sbin/tgt-admin -e
+	end script

Note: I don't tested this patch. I don't know if the post-script stanza runs after a restart operation. It's possible it doesn't work, since the Upstart cookbook says: "Further note that if the job contains post-stop, pre-start or post-start stanzas, these will NOT be run for a restart. However, a pre-stop stanza will be run." See: http://upstart.ubuntu.com/cookbook/#restart

- The `stop tgt` command really stops tgtd even if some LUNs still in use:

-- Initd.sample: Already solved
-- Upstart:

Patch applied to tgt.conf file (debian/upstart, line 11):

+ pre-stop script
+     # Remove all targets. It only removes targets which are not in use.
+ 	/usr/sbin/tgt-admin --update ALL -c /dev/null
+ 	# tgtd will exit if all targets were removed
+ 	/usr/sbin/tgtadm --op delete --mode system
+ 	RETVAL=$?
+ 	if [ "$RETVAL" -ne 0 ] ; then
+ 	    # Some initiators are still connected - could not stop tgtd
+         # TODO Log this message in some place
+ 	    exit 2
+ 	fi
+ end script
+ 
post-start exec /usr/sbin/tgt-admin -e

-----------

This comment is bigger, but it describes well all problems and solutions I found. I hope it helps.

Revision history for this message

Rarylson Freitas (rarylson) wrote on 2018-03-08:

#8

Hi everyone.

It seems like the TGT bug (when stopping, reloading or restarting the TGT deamon) is solved in Ubuntu Artful (17.10). I did not make tests, but now TGT uses the upstream init.d script or the TGT systemd config file (both of them address all of the previous described problems).

The commands `tgtadm --op update --mode sys --name State -v offline`, `tgt-admin --offline ALL` and `tgt-admin -e -c /etc/tgt/targets.conf`, for example, correctly turns TGT down (the to first commands) and reloads in the correct way (the last command).

I think this bug report can be closed now, since it seems to be solved in newer versions.

Below, the systemd tgt service config file:

```
[Unit]
Description=(i)SCSI target daemon
Documentation=man:tgtd(8)
After=network.target

[Service]
Type=notify
TasksMax=infinity
ExecStart=/usr/sbin/tgtd -f
ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline
ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf
ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready

ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline
ExecStop=/usr/sbin/tgt-admin --offline ALL
ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null -f
ExecStop=/usr/sbin/tgtadm --op delete --mode system

ExecReload=/usr/sbin/tgt-admin --update ALL -c /etc/tgt/targets.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-03-12:

#9

Thanks for coming back rarylson!
It seems no one realized this implicitly fixed this issue when the changes were made.
Closing the bug as fix released (since artful).

Changed in tgt (Ubuntu):
status:	Confirmed → Fix Released

Ubuntu
tgt package

tgt doesn't export some LUNs after a reload operation (SIGHUP)

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntutgt package

tgt doesn't export some LUNs after a reload operation (SIGHUP)

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
tgt package