| 2020-05-08 15:55:57 |
Ben Swartzlander |
bug |
|
|
added bug |
| 2020-05-08 18:22:27 |
Ben Swartzlander |
description |
When using iSCSI storage underneath cloud applications such as OpenStack or Kubernetes, the automatic bus scan on login causes problems, because it results in SCSI disks being registered in the kernel that will never get cleaned up, and when those disks are eventually deleted off the server, I/O errors begin to accumulate, eventually slowing down the whole SCSI subsystem, spamming the kernel log, and causing timeouts at higher levels such that users are forced to reboot the node to get back to a usable state.
RedHat discovered this problem more than 3 years ago and fixed it upstream.
https://bugzilla.redhat.com/show_bug.cgi?id=1422941
I had hoped that Debian would eventually pick up the version in which it was fixed, but another LTS has gone by without picking up the newer upstream version, and this is a critical problem, so I propose backporting the fixes.
The 2 patches that need porting are:
https://github.com/open-iscsi/open-iscsi/pull/40
https://github.com/open-iscsi/open-iscsi/pull/49
These changes have been proven safe by 3 years of soak time in the RedHat ecosystem, so I don't see much risk to taking them into Ubuntu. They apply cleanly to the most recent versions of focal, bionic, and xenial. |
[Impact]
When using iSCSI storage underneath cloud applications such as OpenStack or Kubernetes, the automatic bus scan on login causes problems, because it results in SCSI disks being registered in the kernel that will never get cleaned up, and when those disks are eventually deleted off the server, I/O errors begin to accumulate, eventually slowing down the whole SCSI subsystem, spamming the kernel log, and causing timeouts at higher levels such that users are forced to reboot the node to get back to a usable state.
[Test Case]
TBD...
[Regression Potential]
These changes have been proven safe by 3 years of soak time in the RedHat ecosystem, so I don't see much risk to taking them into Ubuntu. They apply cleanly to the most recent versions of focal, bionic, and xenial.
The change introduces a new config option in iscsid.conf but the default is to do exactly what it used to do. Only users who explicitly change this option will get altered behavior, and
[Other Info]
RedHat discovered this problem more than 3 years ago and fixed it upstream.
https://bugzilla.redhat.com/show_bug.cgi?id=1422941
I had hoped that Debian would eventually pick up the version in which it was fixed, but another LTS has gone by without picking up the newer upstream version, and this is a critical problem, so I propose backporting the fixes.
The 2 patches that need porting are:
https://github.com/open-iscsi/open-iscsi/pull/40
https://github.com/open-iscsi/open-iscsi/pull/49 |
|
| 2020-05-08 20:29:24 |
Ben Swartzlander |
description |
[Impact]
When using iSCSI storage underneath cloud applications such as OpenStack or Kubernetes, the automatic bus scan on login causes problems, because it results in SCSI disks being registered in the kernel that will never get cleaned up, and when those disks are eventually deleted off the server, I/O errors begin to accumulate, eventually slowing down the whole SCSI subsystem, spamming the kernel log, and causing timeouts at higher levels such that users are forced to reboot the node to get back to a usable state.
[Test Case]
TBD...
[Regression Potential]
These changes have been proven safe by 3 years of soak time in the RedHat ecosystem, so I don't see much risk to taking them into Ubuntu. They apply cleanly to the most recent versions of focal, bionic, and xenial.
The change introduces a new config option in iscsid.conf but the default is to do exactly what it used to do. Only users who explicitly change this option will get altered behavior, and
[Other Info]
RedHat discovered this problem more than 3 years ago and fixed it upstream.
https://bugzilla.redhat.com/show_bug.cgi?id=1422941
I had hoped that Debian would eventually pick up the version in which it was fixed, but another LTS has gone by without picking up the newer upstream version, and this is a critical problem, so I propose backporting the fixes.
The 2 patches that need porting are:
https://github.com/open-iscsi/open-iscsi/pull/40
https://github.com/open-iscsi/open-iscsi/pull/49 |
[Impact]
When using iSCSI storage underneath cloud applications such as OpenStack or Kubernetes, the automatic bus scan on login causes problems, because it results in SCSI disks being registered in the kernel that will never get cleaned up, and when those disks are eventually deleted off the server, I/O errors begin to accumulate, eventually slowing down the whole SCSI subsystem, spamming the kernel log, and causing timeouts at higher levels such that users are forced to reboot the node to get back to a usable state.
[Test Case]
################
# To demonstrate this problem, I create a VM running Ubuntu 20.04.0
# Install both iSCSI initiator and target on this host
sudo apt-get -y install open-iscsi targetcli-fb
# Start the services
sudo systemctl start iscsid.service targetclid.service
# Create a randomly generated target IQN
TARGET_IQN=$(iscsi-iname)
# Get the initator IQN
INITIATOR_IQN=$(sudo awk -F = '/InitiatorName=/ {print $2}' /etc/iscsi/initiatorname.iscsi)
# Set up an iSCSI target and target portal, and grant access to ourselves
sudo targetcli /iscsi create $TARGET_IQN
sudo targetcli /iscsi/$TARGET_IQN/tpg1/acls create $INITIATOR_IQN
# Create two 1GiB LUNs backed by files, and expose them through the target portal
sudo targetcli /backstores/fileio create lun1 /lun1 1G
sudo targetcli /iscsi/$TARGET_IQN/tpg1/luns create /backstores/fileio/lun1 1
sudo targetcli /backstores/fileio create lun2 /lun2 1G
sudo targetcli /iscsi/$TARGET_IQN/tpg1/luns create /backstores/fileio/lun2 2
# Truncate the kernel log so we can see messages after this point only
sudo dmesg -C
# Register the local iSCSI target with out initiator, and login
sudo iscsiadm -m node -p 127.0.0.1 -T $TARGET_IQN -o new
sudo iscsiadm -m node -p 127.0.0.1 -T $TARGET_IQN --login
# Get the list of disks from the iSCSI session, and stash it in an array
eval "DISKS=\$(sudo iscsiadm -m session -P3 | awk '/Attached scsi disk/ {print \$4}')"
# Print the list
echo $DISKS
# Note that there are two disks found already (the two LUNs we created
# above) despite the fact that we only just logged in.
# Now delete a LUN from the target
sudo targetcli /iscsi/$TARGET_IQN/tpg1/luns delete lun2
sudo targetcli /backstores/fileio delete lun2
# Attempt to read each of the disks
for DISK in $DISKS ; do sudo blkid /dev/$DISK || true ; done
# Look at the kernel log
dmesg
# Notice I/O errors related to the disk that the kernel remembers
################
# Now to demostrate how this problem is fixed, I create a new Ubuntu 20.04.0 VM
# Add PPA with modified version of open-iscsi
sudo add-apt-repository -y ppa:bswartz/open-iscsi
sudo apt-get update
# Install both iSCSI initiator and target on this host
sudo apt-get -y install open-iscsi targetcli-fb
# Start the services
sudo systemctl start iscsid.service targetclid.service
# Set the scan option to "manual"
sudo sed -i 's/^\(node.session.scan\).*/\1 = manual/' /etc/iscsi/iscsid.conf
sudo systemctl restart iscsid.service
# Create a randomly generated target IQN
TARGET_IQN=$(iscsi-iname)
# Get the initator IQN
INITIATOR_IQN=$(sudo awk -F = '/InitiatorName=/ {print $2}' /etc/iscsi/initiatorname.iscsi)
# Set up an iSCSI target and target portal, and grant access to ourselves
sudo targetcli /iscsi create $TARGET_IQN
sudo targetcli /iscsi/$TARGET_IQN/tpg1/acls create $INITIATOR_IQN
# Create two 1GiB LUNs backed by files, and expose them through the target portal
sudo targetcli /backstores/fileio create lun1 /lun1 1G
sudo targetcli /iscsi/$TARGET_IQN/tpg1/luns create /backstores/fileio/lun1 1
sudo targetcli /backstores/fileio create lun2 /lun2 1G
sudo targetcli /iscsi/$TARGET_IQN/tpg1/luns create /backstores/fileio/lun2 2
# Truncate the kernel log so we can see messages after this point only
sudo dmesg -C
# Register the local iSCSI target with out initiator, and login
sudo iscsiadm -m node -p 127.0.0.1 -T $TARGET_IQN -o new
sudo iscsiadm -m node -p 127.0.0.1 -T $TARGET_IQN --login
# Get the list of disks from the iSCSI session, and stash it in an array
eval "DISKS=\$(sudo iscsiadm -m session -P3 | awk '/Attached scsi disk/ {print \$4}')"
# Print the list
echo $DISKS
# Note that the list is empty!
# Get the iSCSI host
SCSI_HOST=$(ls /sys/class/iscsi_host)
# Specifically scan the one disk we want
sudo sh -c "echo '0 0 1' > /sys/class/scsi_host/$SCSI_HOST/scan"
# Get the list of disks from the iSCSI session, and stash it in an array
eval "DISKS=\$(sudo iscsiadm -m session -P3 | awk '/Attached scsi disk/ {print \$4}')"
# Print the list
echo $DISKS
# This time notice there's exactly one disk
# Now delete the other LUN from the target
sudo targetcli /iscsi/$TARGET_IQN/tpg1/luns delete lun2
sudo targetcli /backstores/fileio delete lun2
# Attempt to read each of the disks
for DISK in $DISKS ; do sudo blkid /dev/$DISK || true ; done
# Look at the kernel log
dmesg
# No errors in the log
################
[Regression Potential]
These changes have been proven safe by 3 years of soak time in the RedHat ecosystem, so I don't see much risk to taking them into Ubuntu. They apply cleanly to the most recent versions of focal, bionic, and xenial.
The change introduces a new config option in iscsid.conf but the default is to do exactly what it used to do. Only users who explicitly change this option will get altered behavior, and the behavior with the option set is
superior for the above mentioned cloud use cases.
[Other Info]
RedHat discovered this problem more than 3 years ago and fixed it upstream.
https://bugzilla.redhat.com/show_bug.cgi?id=1422941
I had hoped that Debian would eventually pick up the version in which it was fixed, but another LTS has gone by without picking up the newer upstream version, and this is a critical problem, so I propose backporting the fixes.
The 2 patches that need porting are:
https://github.com/open-iscsi/open-iscsi/pull/40
https://github.com/open-iscsi/open-iscsi/pull/49 |
|
| 2020-05-09 00:54:15 |
Ben Swartzlander |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
bug task added |
|
open-iscsi (Ubuntu Groovy) |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
nominated for series |
|
Ubuntu Bionic |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
bug task added |
|
open-iscsi (Ubuntu Bionic) |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
nominated for series |
|
Ubuntu Focal |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
bug task added |
|
open-iscsi (Ubuntu Focal) |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
nominated for series |
|
Ubuntu Eoan |
|
| 2020-05-10 22:01:29 |
Rafael David Tinoco |
bug task added |
|
open-iscsi (Ubuntu Eoan) |
|
| 2020-05-10 22:01:39 |
Rafael David Tinoco |
open-iscsi (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2020-05-10 22:01:41 |
Rafael David Tinoco |
open-iscsi (Ubuntu Eoan): status |
New |
Confirmed |
|
| 2020-05-10 22:01:43 |
Rafael David Tinoco |
open-iscsi (Ubuntu Focal): status |
New |
Confirmed |
|
| 2020-05-10 22:01:45 |
Rafael David Tinoco |
open-iscsi (Ubuntu Groovy): status |
New |
Confirmed |
|
| 2020-05-10 22:01:48 |
Rafael David Tinoco |
open-iscsi (Ubuntu Groovy): assignee |
|
Rafael David Tinoco (rafaeldtinoco) |
|
| 2020-05-10 22:01:56 |
Rafael David Tinoco |
bug |
|
|
added subscriber Ubuntu Server |
| 2020-05-10 22:03:00 |
Rafael David Tinoco |
tags |
|
block-proposed-focal block-proposed-groovy |
|
| 2020-05-11 01:37:37 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/open-iscsi/+git/open-iscsi/+merge/383682 |
|
| 2020-05-11 01:58:02 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/open-iscsi/+git/open-iscsi/+merge/383683 |
|
| 2020-05-11 02:09:20 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/open-iscsi/+git/open-iscsi/+merge/383684 |
|
| 2020-05-11 02:11:44 |
Rafael David Tinoco |
open-iscsi (Ubuntu Groovy): status |
Confirmed |
Triaged |
|
| 2020-05-11 02:11:46 |
Rafael David Tinoco |
open-iscsi (Ubuntu Focal): status |
Confirmed |
In Progress |
|
| 2020-05-11 02:11:48 |
Rafael David Tinoco |
open-iscsi (Ubuntu Eoan): status |
Confirmed |
In Progress |
|
| 2020-05-11 02:11:50 |
Rafael David Tinoco |
open-iscsi (Ubuntu Bionic): status |
Confirmed |
In Progress |
|
| 2020-05-11 02:15:52 |
Rafael David Tinoco |
bug |
|
|
added subscriber Ritesh Raj Sarraf |
| 2020-05-11 02:16:23 |
Rafael David Tinoco |
summary |
Automatic scans cause instability for cloud use cases |
open-iscsi automatic scans disable AND groovy merge with upstream |
|
| 2020-05-11 02:26:24 |
Mathew Hodson |
open-iscsi (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2020-05-11 02:26:26 |
Mathew Hodson |
open-iscsi (Ubuntu Eoan): importance |
Undecided |
Medium |
|
| 2020-05-11 02:26:29 |
Mathew Hodson |
open-iscsi (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2020-05-11 02:26:32 |
Mathew Hodson |
open-iscsi (Ubuntu Groovy): importance |
Undecided |
Medium |
|
| 2020-05-12 15:42:07 |
Mathew Hodson |
bug |
|
|
added subscriber Mathew Hodson |
| 2020-05-20 01:08:22 |
Mathew Hodson |
summary |
open-iscsi automatic scans disable AND groovy merge with upstream |
Automatic scans cause instability for cloud use cases |
|
| 2020-05-22 22:15:57 |
Steve Langasek |
open-iscsi (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2020-05-22 22:16:01 |
Steve Langasek |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-05-22 22:16:05 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2020-05-22 22:16:10 |
Steve Langasek |
tags |
block-proposed-focal block-proposed-groovy |
block-proposed-focal block-proposed-groovy verification-needed verification-needed-focal |
|
| 2020-05-22 22:23:30 |
Steve Langasek |
open-iscsi (Ubuntu Eoan): status |
In Progress |
Fix Committed |
|
| 2020-05-22 22:23:35 |
Steve Langasek |
tags |
block-proposed-focal block-proposed-groovy verification-needed verification-needed-focal |
block-proposed-focal block-proposed-groovy verification-needed verification-needed-eoan verification-needed-focal |
|
| 2020-05-22 22:25:15 |
Steve Langasek |
open-iscsi (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2020-05-22 22:25:23 |
Steve Langasek |
tags |
block-proposed-focal block-proposed-groovy verification-needed verification-needed-eoan verification-needed-focal |
block-proposed-focal block-proposed-groovy verification-needed verification-needed-bionic verification-needed-eoan verification-needed-focal |
|
| 2020-05-22 22:25:57 |
Steve Langasek |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2020-05-23 03:38:16 |
Mathew Hodson |
removed subscriber Mathew Hodson |
|
|
|
| 2020-05-26 03:55:52 |
Rafael David Tinoco |
tags |
block-proposed-focal block-proposed-groovy verification-needed verification-needed-bionic verification-needed-eoan verification-needed-focal |
verification-done verification-done-bionic verification-done-eoan verification-done-focal |
|
| 2020-05-26 18:52:23 |
Rafael David Tinoco |
nominated for series |
|
Ubuntu Xenial |
|
| 2020-05-26 18:52:23 |
Rafael David Tinoco |
bug task added |
|
open-iscsi (Ubuntu Xenial) |
|
| 2020-05-26 18:52:32 |
Rafael David Tinoco |
open-iscsi (Ubuntu Xenial): status |
New |
Won't Fix |
|
| 2020-05-26 18:52:39 |
Rafael David Tinoco |
open-iscsi (Ubuntu Groovy): assignee |
Rafael David Tinoco (rafaeldtinoco) |
|
|
| 2020-06-02 16:48:24 |
Launchpad Janitor |
open-iscsi (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2020-06-02 16:48:30 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-06-02 16:48:47 |
Launchpad Janitor |
open-iscsi (Ubuntu Eoan): status |
Fix Committed |
Fix Released |
|
| 2020-06-02 16:49:08 |
Launchpad Janitor |
open-iscsi (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2020-06-19 02:26:00 |
Rafael David Tinoco |
bug task deleted |
open-iscsi (Ubuntu Groovy) |
|
|
| 2020-06-19 02:26:40 |
Rafael David Tinoco |
open-iscsi (Ubuntu): status |
Triaged |
Fix Released |
|
