[Ubuntu 22.04 LTS]The host OS becomes read-only after the 'rescan-scsi-bus.sh -r' command is executed

[zihao yang]

[Impact]

Users are not able to use iscsi devices after executing the 'rescan-scsi-bus.sh' script.

[Test Plan]

In a system with iscsi devices properly configured run the following:

# Make sure all block devices are in place and as expected
$ lsblk
# Execute the script
$ rescan-scsi-bus.sh -r
# Check again if all block devices are still there
$ lsblk

In the current buggy version of this script, you will not see the block devices representing the iscsi device after executing it. With a fixed version, all of them should still be there.

[Where problems could occur]

The fix basically adjust the way the output of sg_inq is parsed (the structure of the output changed and that's the issue), a problem that could be faced here is in the case the user has another version of the sg_inq command that differs from the expected output. Then the proposed fix could or couldn't work.

[Original description]

OS:Ubuntu 22.04 LTS
kernel:5.15.0-33-generic
host model:Dell PowerEdge R630
issue description:
After I mapped luns from array to my host, I ran 'rescan-scsi-bus.sh -r' to scan for luns, my host crashed and became read-only.After the KVM forcibly restarts the host, the host is restored to normal.
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 30 08:33 seq
 crw-rw---- 1 root audio 116, 33 May 30 08:33 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2022-05-26 (4 days ago)
InstallationMedia: Ubuntu-Server 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220421)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-33-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
ProcVersionSignature: Ubuntu 5.15.0-33.34-generic 5.15.30
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-33-generic N/A
 linux-backports-modules-5.15.0-33-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-33-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 05/14/2021
dmi.bios.release: 2.13
dmi.bios.version: 2.13.0
dmi.board.name: 02C2CP
dmi.board.version: A01
dmi.chassis.type: 23
dmi.modalias: dmi:bvn:bvr2.13.0:bd05/14/2021:br2.13:svn:pn:pvr:rvn:rn02C2CP:rvrA01:cvn:ct23:cvr:skuSKU=NotProvided;ModelName=:
dmi.product.sku: SKU=NotProvided;ModelName=

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[zihao yang (zihao55)]

apport information

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[vincent chen (cheny38)]

Canional team, is there any update this ticket?

[zihao yang (zihao55)]

summary of issues:
1.“rescan-scsi-bus.sh -r” remove local disks wrongly

Scanning host 5 for all SCSI target IDs, all LUNs
 Scanning for device 5 0 0 0 ..sg0 changed: LU not available (PQual 3)
REM: Host: scsi5 Channel: 00 Id: 00 Lun: 00
DEL: Vendor: EMC Model: LUNZ Rev: 7010
      Type: Direct-Access ANSI SCSI revision: 04
Scanning host 6 for all SCSI target IDs, all LUNs
 Scanning for device 6 0 0 0 ..sg1 changed: LU not available (PQual 3)
REM: Host: scsi6 Channel: 00 Id: 00 Lun: 00
DEL: Vendor: EMC Model: LUNZ Rev: 7010
      Type: Direct-Access ANSI SCSI revision: 04
Scanning host 7 for all SCSI target IDs, all LUNs
Scanning host 8 for all SCSI target IDs, all LUNs
Scanning host 9 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
0 new or changed device(s) found.
0 remapped or resized device(s) found.
6 device(s) removed.

2.the root lv was stopped by systemd after “rescan-scsi-bus.sh -r”

Jul 14 02:54:47 ubuntu2204 kernel: [1976751.752137] blk_update_request: I/O error, dev sdd, sector 707289518 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
Jul 14 02:54:47 ubuntu2204 kernel: [1976751.752534] blk_update_request: I/O error, dev sdd, sector 704106197 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 0
… …
Jul 14 02:54:47 ubuntu2204 systemd[1]: Stopping LVM event activation on device 8:51...
Jul 14 02:54:47 ubuntu2204 systemd[1]: Stopping Unattended Upgrades Shutdown...
… …
Jul 14 02:54:47 ubuntu2204 systemd[1]: Stopped target Local File Systems.
Jul 14 02:54:47 ubuntu2204 systemd[1]: Unmounting /boot...

3.whole / mount status change to ro after “rescan-scsi-bus.sh -r”

/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (ro,relatime)

notes:
This issue was reproduced on different hosts,and it is totally reproducible.We did test on host with local disk only.Please refer to the reproduction below.

issue repetition:
OS details:
root@ubuntu2204:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@ubuntu2204:~# uname -r
5.15.0-39-generic

'fdisk -l' output:
root@ubuntu2204:~# fdisk -l
Disk /dev/loop0: 61.92 MiB, 64925696 bytes, 126808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop1: 61.93 MiB, 64933888 bytes, 126824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop2: 79.95 MiB, 83832832 bytes, 163736 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop3: 46.95 MiB, 49233920 bytes, 96160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 byt...

[Michael Reed (mreed8855)]

Is this an issue on 20.04 with the 5.4 kernel or on 22.10?

[Jeff Lane  (bladernr)]

Also is this reproducible on anything modern (like 14 or 15G hardware?)
What is the config of this system?
What is the storage set up like?

[zihao yang (zihao55)]

Hello Jeff,
The reproducible on Dell PowerEdge R630.The test system is initialized system(OS info:Ubuntu 22.04 LTS
kernel:5.15.0-33-generic) without any special configiration.

[zihao yang (zihao55)]

Hello,
Are there any updates for this bug?I retest this in the lasted OS(Ubuntu 22.04.1 LTS kernel:5.15.0-43-generic),The issue remains,I uploaded the details of the reenactment and the system logs,Please help to check.

[Jeff Lane  (bladernr)]

The R630 is no longer sold by Dell and has not been certifed since 16.04. While usually these continue working fine with OS versions newer than the one that was certified, there is no guarantee that will always be the case.

What is driving this?

[zihao yang (zihao55)]

Hello jeff,
Thanks for the update.I will confirm the infomation that you mentioned.However,For the last test I used a DELL 15G server(Dell PowerEdge R6515),the issue still exists,Is this model not certified either?

[zihao yang (zihao55)]

Any update for this issue?

[zihao yang (zihao55)]

Hello jeff,
This seems unrelated with external storage devices, I did the test on host with local disk only,
“rescan-scsi-bus.sh -r” remove local disks wrongly.
After excuting “rescan-scsi-bus.sh -r” ,whole / mount status change to ro:
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (ro,relatime)

I hope this information will help you locate the issue,and find a solution to the script.

Br,
Zihao Yang

[Jeff Lane  (bladernr)]

Moved this because it seems to be an issue specifically with rescan_scsi_bus.sh... it may be that there IS a kernel component too, but for now I was able to obliterate a system using "rescan-scsi-bus.sh -r" on a system wiht a pretty simple local FS setup:

ubuntu@barbos:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32718576k,nr_inodes=8179644,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6555388k,mode=755,inode64)
/dev/sda2 on / type ext4 (rw,relatime,stripe=128)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=131329)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/qemu type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
/var/lib/snapd/snaps/lxd_24322.snap on /snap/lxd/24322 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_18357.snap on /snap/snapd/18357 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdb1 on /data-sdb type ext4 (rw,relatime,stripe=64) ...

[Jeff Lane  (bladernr)]

Retried and checked dmesg while the carnage was ensuing, this is the last, dying gasp of my server:
[ 275.205762] systemd-journald[1593]: File /var/log/journal/c06e1eb8d79e4afea01453d6a9680f1b/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.
[ 315.556607] blk_update_request: I/O error, dev sda, sector 1578109256 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[ 315.556608] blk_update_request: I/O error, dev sda, sector 34997584 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556618] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20715819 starting block 4374699)
[ 315.556697] Buffer I/O error on dev sda2, logical block 197132329, lost async page write
[ 315.556743] blk_update_request: I/O error, dev sda, sector 34354704 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556754] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294339)
[ 315.556762] Buffer I/O error on device sda2, logical block 4163010
[ 315.556770] blk_update_request: I/O error, dev sda, sector 34355552 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556776] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294445)
[ 315.556780] Buffer I/O error on device sda2, logical block 4163116
[ 315.556783] Buffer I/O error on device sda2, logical block 4243370
[ 315.556784] blk_update_request: I/O error, dev sda, sector 34355616 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556789] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294453)
[ 315.556793] Buffer I/O error on device sda2, logical block 4163124
[ 315.556796] blk_update_request: I/O error, dev sda, sector 34355832 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556800] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294480)
[ 315.556803] Buffer I/O error on device sda2, logical block 4163151
[ 315.556810] blk_update_request: I/O error, dev sda, sector 34356944 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556813] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294619)
[ 315.556817] Buffer I/O error on device sda2, logical block 4163290
[ 315.556820] blk_update_request: I/O error, dev sda, sector 34357672 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556823] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294710)
[ 315.556826] Buffer I/O error on device sda2, logical block 4163381
[ 315.556829] blk_update_request: I/O error, dev sda, sector 34358520 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 315.556832] EXT4-fs warning (device sda2): ext4_end_bio:344: I/O error 10 writing to inode 20718352 starting block 4294820)
[ 315.556835] Buffer I/O error on device sda2, logical block 4163487
[ 315.556837] Buffer I/O error on device sda2, logical block 4163488
[ 315.556916] blk_update_request: I/O error, dev sda, sector 1066404192 op 0x1:(WRITE) flags 0x103000 ph...

[Trent Lloyd (lathiat)]

This fixed which is merged upstream in Jan 2022 seems like a likely candidate fix:
https://github.com/doug-gilbert/sg3_utils/pull/10

System was crashing when 'rescan-scsi-bus.sh -r' was run. This is because of the changes in the output of sg_inq command. In it, "Device_type" has been replaced with "PDT". So when Device_type is used in sed, it was passing incorrect return value to dolunscan() and hence even the internal disk from which system is booting was getting deleted. Mentioned fix takes care of both the outputs of sg_inq command. Only in recent sg_inq we see "PDT". Earlier sg_inq still shows "Device_type".

[Jeff Lane  (bladernr)]

Testing that PR... this is the result on bare metal using a patched version of rescan-scsi-bus.sh:

ubuntu@barbos:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.3M 1 loop /snap/core20/1828
loop1 7:1 0 63.3M 1 loop /snap/core20/1852
loop2 7:2 0 111.9M 1 loop /snap/lxd/24322
loop3 7:3 0 49.8M 1 loop /snap/snapd/18357
loop4 7:4 0 49.8M 1 loop /snap/snapd/18596
sda 8:0 0 800G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
└─sda2 8:2 0 799.5G 0 part /
sdb 8:16 0 438.3G 0 disk
└─sdb1 8:17 0 438.3G 0 part /data-sdb
ubuntu@barbos:~$ sudo ./rescan-scsi-bus.sh -r
Syncing file systems
Scanning SCSI subsystem for new devices and remove devices that have disappeared
Scanning host 0 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
 Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: HPE Model: Smart Adapter Rev: 3.53
      Type: Enclosure ANSI SCSI revision: 05
 Scanning for device 0 2 0 0 ... 6 ...
OLD: Host: scsi0 Channel: 02 Id: 00 Lun: 00
      Vendor: HPE Model: P408i-a SR Gen10 Rev: 3.53
      Type: RAID ANSI SCSI revision: 05
 Scanning for device 0 1 0 0 ... 6 ...
OLD: Host: scsi0 Channel: 01 Id: 00 Lun: 00
      Vendor: HPE Model: LOGICAL VOLUME Rev: 3.53
      Type: Direct-Access ANSI SCSI revision: 05
 Scanning for device 0 1 0 1 ...
OLD: Host: scsi0 Channel: 01 Id: 00 Lun: 01
      Vendor: HPE Model: LOGICAL VOLUME Rev: 3.53
      Type: Direct-Access ANSI SCSI revision: 05
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.
ubuntu@barbos:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.3M 1 loop /snap/core20/1828
loop1 7:1 0 63.3M 1 loop /snap/core20/1852
loop2 7:2 0 111.9M 1 loop /snap/lxd/24322
loop3 7:3 0 49.8M 1 loop /snap/snapd/18357
loop4 7:4 0 49.8M 1 loop /snap/snapd/18596
sda 8:0 0 800G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
└─sda2 8:2 0 799.5G 0 part /
sdb 8:16 0 438.3G 0 disk
└─sdb1 8:17 0 438.3G 0 part /data-sdb

[Jeff Lane  (bladernr)]

And this is with the current version in Jammy (broken version) just for comparison, it obviously fails...

ubuntu@barbos:~$ sudo rescan-scsi-bus.sh -r
Syncing file systems
Scanning SCSI subsystem for new devices and remove devices that have disappeared
Scanning host 0 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
sg0 changed: LU not available (PQual 3)
REM: Host: scsi0 Channel: 00 Id: 00 Lun: 00
DEL: Vendor: HPE Model: Smart Adapter Rev: 3.53
      Type: Enclosure ANSI SCSI revision: 05
sg3 changed: LU not available (PQual 3)
REM: Host: scsi0 Channel: 02 Id: 00 Lun: 00
DEL: Vendor: HPE Model: P408i-a SR Gen10 Rev: 3.53
      Type: RAID ANSI SCSI revision: 05
sg1 changed: LU not available (PQual 3)
REM: Host: scsi0 Channel: 01 Id: 00 Lun: 00
      Vendor: HPE Model: LOGICAL VOLUME Rev: 3.53
DEL: Type: Direct-Access ANSI SCSI revision: 05

 Scanning for device 0 1 0 1 ... 535: /usr/bin/sleep: Input/output error
OLD: /usr/bin/rescan-scsi-bus.sh: line 153: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 154: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 155: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 161: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 162: cat: command not found
OLD: /usr/bin/rescan-scsi-bus.sh: line 153: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 154: cat: command not foundd
/usr/bin/rescan-scsi-bus.sh: line 155: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 161: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 162: cat: command not found
/usr/bin/rescan-scsi-bus.sh: line 357: head: command not found
/usr/bin/rescan-scsi-bus.sh: line 358: tail: command not found
/usr/bin/rescan-scsi-bus.sh: line 358: pr: command not found
0 new or changed device(s) found.
0 remapped or resized device(s) found.
3 device(s) removed.
\t[0:0:0:0]\n\t[0lsblk
-bash: /usr/bin/lsblk: Input/output error

[Lucas Kanashiro (lucaskanashiro)]

I prepared a PPA applying the patch mentioned in comment #30 here:

https://launchpad.net/~lucaskanashiro/+archive/ubuntu/rescan-scsi-bus-fix/

If people report back saying this package fixes the issue we can work on a SRU. FWIW we have a similar issue in this other bug report:

https://bugs.launchpad.net/ubuntu/+source/sg3-utils/+bug/1966860

[Jeff Lane  (bladernr)]

Checked Bionic and it does not fail for me like it does on Jammy.

[zihao yang (zihao55)]

Thanks for the update,
I've done regression testing with this fix packages:
https://launchpad.net/~lucaskanashiro/+archive/ubuntu/rescan-scsi-bus-fix/.
And I can confirm that the fix has taken effect and resolved the issue,I think this bug could be closed.
Please check attached documentation for regression testing procedures.
Regards,
Zihao Yang

[Jeff Lane  (bladernr)]

Thank you for confirming the fix works. Lets now look at getting this SRUd into 22.04 and later.

[Lucas Kanashiro (lucaskanashiro)]

I am starting the SRU process to land this fix in Jammy and Focal but I will need help from you guys to verify the fix once it lands in -proposed. Stay tuned!

[Andreas Hasenack (ahasenack)]

I believe this does not affect focal, because in focal, sg_inq returns Device_type, and not PDT, which means the fix is not needed there.

I can reproduce it with a VM to which I attach storage as scsi device. If I run "rescan-scsi-bus.sh -r", that device will be removed. And "sg_inq /dev/sda" (assuming the scsi device got named sda) will show PDT= instead of Device_type= in its output, which is what triggers this bug.

focal:
1.44-1ubuntu2
root@f-sg3:~# sg_inq /dev/sda | grep -E "(PDT|Device_type)"
  PQual=0 Device_type=0 RMB=0 LU_CONG=0 version=0x05 [SPC-3]

root@f-sg3:~# ll /dev/sda
brw-rw---- 1 root disk 8, 0 Sep 5 19:26 /dev/sda

root@f-sg3:~# rescan-scsi-bus.sh -r
Syncing file systems
(...)
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.

root@f-sg3:~# ll /dev/sda
brw-rw---- 1 root disk 8, 0 Sep 5 19:26 /dev/sda

So focal seems fine.

But I see the problem in lunar and mantic still (besides jammy).

Lunar:
1.46-1build1
root@l-sg3:~# sg_inq /dev/sda | grep -E "(PDT|Device_type)"
  PQual=0 PDT=0 RMB=0 LU_CONG=0 hot_pluggable=0 version=0x05 [SPC-3]

root@l-sg3:~# ll /dev/sda
brw-rw---- 1 root disk 8, 0 Sep 5 19:23 /dev/sda

root@l-sg3:~# rescan-scsi-bus.sh -r
Syncing file systems
(...)
0 remapped or resized device(s) found.
1 device(s) removed.

root@l-sg3:~# ll /dev/sda
ls: cannot access '/dev/sda': No such file or directory

And the same for mantic.

[Timo Aaltonen (tjaalton)]

this doesn't seem to be in mantic yet, so uploading SRU's was premature?

[Steve Langasek (vorlon)]

This has been uploaded into the lunar SRU queue but I don't see the corresponding change anywhere yet for mantic.

[Lucas Kanashiro (lucaskanashiro)]

Sorry, I just uploaded the fix to Mantic.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sg3-utils - 1.46-3ubuntu2

---------------
sg3-utils (1.46-3ubuntu2) mantic; urgency=medium

  * d/p/0002-Fix-crash-with-rescan-scsi-bus.sh-r.patch: do not allow
    rescan-scsi-bus.sh to remove device (LP: #1976228).

 -- Lucas Kanashiro <email address hidden> Tue, 05 Sep 2023 18:22:04 -0300

[Robie Basak (racb)]

Hello zihao, or anyone else affected,

Accepted sg3-utils into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sg3-utils/1.46-1ubuntu0.23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello zihao, or anyone else affected,

Accepted sg3-utils into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sg3-utils/1.46-1ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lucas Kanashiro (lucaskanashiro)]

# Verification Jammy

ubuntu@ha-agent-virsh-jammy-sru1-node01:~$ sudo sg_inq /dev/sda | grep -E "(PDT|Device_type)"
  PQual=0 PDT=0 RMB=0 LU_CONG=0 hot_pluggable=0 version=0x06 [SPC-4]
ubuntu@ha-agent-virsh-jammy-sru1-node01:~$ ll /dev/sda
brw-rw---- 1 root disk 8, 0 Oct 5 20:24 /dev/sda
ubuntu@ha-agent-virsh-jammy-sru1-node01:~$ dpkg -l sg3-utils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=====================-============-================================================
ii sg3-utils 1.46-1ubuntu0.22.04.1 amd64 utilities for devices using the SCSI command set
ubuntu@ha-agent-virsh-jammy-sru1-node01:~$ sudo rescan-scsi-bus.sh -r
Syncing file systems
Scanning SCSI subsystem for new devices and remove devices that have disappeared
Scanning host 0 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 1 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 2 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 3 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 4 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 5 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 6 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
 Scanning for device 6 0 0 0 ...
OLD: Host: scsi6 Channel: 00 Id: 00 Lun: 00
      Vendor: LIO-ORG Model: iscsi-disk01 Rev: 4.0
      Type: Direct-Access ANSI SCSI revision: 06
Scanning host 7 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
 Scanning for device 7 0 0 0 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 00
      Vendor: LIO-ORG Model: iscsi-disk01 Rev: 4.0
      Type: Direct-Access ANSI SCSI revision: 06
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.
ubuntu@ha-agent-virsh-jammy-sru1-node01:~$ ll /dev/sda
brw-rw---- 1 root disk 8, 0 Oct 5 20:24 /dev/sda

[Lucas Kanashiro (lucaskanashiro)]

# Verification lunar

ubuntu@ha-agent-virsh-lunar-sru1-node01:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=lunar
LOGO=ubuntu-logo
ubuntu@ha-agent-virsh-lunar-sru1-node01:~$ sudo sg_inq /dev/sda | grep -E "(PDT|Device_type)"
  PQual=0 PDT=0 RMB=0 LU_CONG=0 hot_pluggable=0 version=0x06 [SPC-4]
ubuntu@ha-agent-virsh-lunar-sru1-node01:~$ ll /dev/sda
brw-rw---- 1 root disk 8, 0 Oct 5 21:59 /dev/sda
ubuntu@ha-agent-virsh-lunar-sru1-node01:~$ dpkg -l sg3-utils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=====================-============-================================================
ii sg3-utils 1.46-1ubuntu0.23.04.1 amd64 utilities for devices using the SCSI command set
ubuntu@ha-agent-virsh-lunar-sru1-node01:~$ sudo rescan-scsi-bus.sh -r
Syncing file systems
Scanning SCSI subsystem for new devices and remove devices that have disappeared
Scanning host 0 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 1 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 2 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 3 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 4 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 5 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
Scanning host 6 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
 Scanning for device 6 0 0 0 ...
OLD: Host: scsi6 Channel: 00 Id: 00 Lun: 00
      Vendor: LIO-ORG Model: iscsi-disk01 Rev: 4.0
      Type: Direct-Access ANSI SCSI revision: 06
Scanning host 7 for SCSI target IDs 0
1
2
3
4
5
6
7, all LUNs
 Scanning for device 7 0 0 0 ...
OLD: Host: scsi7 Channel: 00 Id: 00 Lun: 00
      Vendor: LIO-ORG Model: iscsi-disk01 Rev: 4.0
      Type: Direct-Access ANSI SCSI revision: 06
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.
ubuntu@ha-agent-virsh-lunar-sru1-node01:~$ ll /dev/sda
brw-rw---- 1 root disk 8, 0 Oct 5 21:59 /dev/sda

[vmware-gos-Yuhua (yhzou)]

Hi Lucas Kanashiro,
1. Passed to verify sg3-utils https://launchpad.net/ubuntu/+source/sg3-utils/1.46-1ubuntu0.22.04.1 from jammy-proposed repo.

2. It won't get the package https://launchpad.net/ubuntu/+source/sg3-utils/1.46-1ubuntu0.23.04.1 from lunar-proposed. Please see details as below:

root@ubuntu:/home/vmware# cat /etc/os-release
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=lunar
LOGO=ubuntu-logo

root@ubuntu:/home/vmware# apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu lunar InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu lunar-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu lunar-backports InRelease
Get:4 http://us.archive.ubuntu.com/ubuntu lunar-proposed InRelease [255 kB]
Hit:5 http://us.archive.ubuntu.com/ubuntu lunar-security InRelease
Get:6 http://us.archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages [127 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu lunar-proposed/main Translation-en [33.1 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu lunar-proposed/main amd64 c-n-f Metadata [2,528 B]
Get:9 http://us.archive.ubuntu.com/ubuntu lunar-proposed/restricted amd64 Packages [138 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu lunar-proposed/restricted Translation-en [25.9 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu lunar-proposed/restricted amd64 c-n-f Metadata [348 B]
Get:12 http://us.archive.ubuntu.com/ubuntu lunar-proposed/universe amd64 Packages [42.2 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu lunar-proposed/universe Translation-en [16.5 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu lunar-proposed/universe amd64 c-n-f Metadata [1,608 B]
Get:15 http://us.archive.ubuntu.com/ubuntu lunar-proposed/multiverse amd64 Packages [748 B]
Get:16 http://us.archive.ubuntu.com/ubuntu lunar-proposed/multiverse Translation-en [660 B]
Get:17 http://us.archive.ubuntu.com/ubuntu lunar-proposed/multiverse amd64 c-n-f Metadata [116 B]
Fetched 644 kB in 2s (359 kB/s)
Reading package lists...

root@ubuntu:/home/vmware# apt-get install sg3-utils
Reading package lists...
Building dependency tree...
Reading state information...
sg3-utils is already the newest version (1.46-1build1).
sg3-utils set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.

root@ubuntu:/home/vmware# dpkg -l sg3-utils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-================================================
ii sg3-utils 1.46-1build1 amd64 utilities for devices using the SCSI command set

Best regards
Yuhua Zou

[Andreas Hasenack (ahasenack)]

Hi @yhzou,

you need to specify -t lunar-proposed, like this:

apt-get install sg3-utils -t lunar-proposed

[Launchpad Janitor (janitor)]

This bug was fixed in the package sg3-utils - 1.46-1ubuntu0.23.04.1

---------------
sg3-utils (1.46-1ubuntu0.23.04.1) lunar; urgency=medium

  * d/p/0002-Fix-crash-with-rescan-scsi-bus.sh-r.patch: do not allow
    rescan-scsi-bus.sh to remove device (LP: #1976228).

 -- Lucas Kanashiro <email address hidden> Tue, 05 Sep 2023 18:14:57 -0300

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for sg3-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sg3-utils - 1.46-1ubuntu0.22.04.1

---------------
sg3-utils (1.46-1ubuntu0.22.04.1) jammy; urgency=medium

  * d/p/0002-Fix-crash-with-rescan-scsi-bus.sh-r.patch: do not allow
    rescan-scsi-bus.sh to remove device (LP: #1976228).

 -- Lucas Kanashiro <email address hidden> Wed, 22 Mar 2023 11:57:45 -0300

[vmware-gos-Yuhua (yhzou)]

> Hi @yhzou,

> you need to specify -t lunar-proposed, like this:

>. apt-get install sg3-utils -t lunar-proposed

Hi Andreas Hasenack,

Thanks.
Passed to verify the fix in ubuntu 23.04 live server.

Best regards
Yuhua Zou