summary of issues: 1.“rescan-scsi-bus.sh -r” remove local disks wrongly Scanning host 5 for all SCSI target IDs, all LUNs Scanning for device 5 0 0 0 ..sg0 changed: LU not available (PQual 3) REM: Host: scsi5 Channel: 00 Id: 00 Lun: 00 DEL: Vendor: EMC Model: LUNZ Rev: 7010 Type: Direct-Access ANSI SCSI revision: 04 Scanning host 6 for all SCSI target IDs, all LUNs Scanning for device 6 0 0 0 ..sg1 changed: LU not available (PQual 3) REM: Host: scsi6 Channel: 00 Id: 00 Lun: 00 DEL: Vendor: EMC Model: LUNZ Rev: 7010 Type: Direct-Access ANSI SCSI revision: 04 Scanning host 7 for all SCSI target IDs, all LUNs Scanning host 8 for all SCSI target IDs, all LUNs Scanning host 9 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs 0 new or changed device(s) found. 0 remapped or resized device(s) found. 6 device(s) removed. 2.the root lv was stopped by systemd after “rescan-scsi-bus.sh -r” Jul 14 02:54:47 ubuntu2204 kernel: [1976751.752137] blk_update_request: I/O error, dev sdd, sector 707289518 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0 Jul 14 02:54:47 ubuntu2204 kernel: [1976751.752534] blk_update_request: I/O error, dev sdd, sector 704106197 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 0 … … Jul 14 02:54:47 ubuntu2204 systemd[1]: Stopping LVM event activation on device 8:51... Jul 14 02:54:47 ubuntu2204 systemd[1]: Stopping Unattended Upgrades Shutdown... … … Jul 14 02:54:47 ubuntu2204 systemd[1]: Stopped target Local File Systems. Jul 14 02:54:47 ubuntu2204 systemd[1]: Unmounting /boot... 3.whole / mount status change to ro after “rescan-scsi-bus.sh -r” /dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (ro,relatime) notes: This issue was reproduced on different hosts,and it is totally reproducible.We did test on host with local disk only.Please refer to the reproduction below. issue repetition: OS details: root@ubuntu2204:~# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy root@ubuntu2204:~# uname -r 5.15.0-39-generic 'fdisk -l' output: root@ubuntu2204:~# fdisk -l Disk /dev/loop0: 61.92 MiB, 64925696 bytes, 126808 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/loop1: 61.93 MiB, 64933888 bytes, 126824 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/loop2: 79.95 MiB, 83832832 bytes, 163736 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/loop3: 46.95 MiB, 49233920 bytes, 96160 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/loop4: 46.96 MiB, 49242112 bytes, 96176 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sde: 1.45 TiB, 1599741100032 bytes, 3124494336 sectors Disk model: PERC H730 Mini Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/sdc: 372 GiB, 399431958528 bytes, 780140544 sectors Disk model: PERC H730 Mini Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 713F881C-1020-4C8A-B679-38DCFE17B2C7 Device Start End Sectors Size Type /dev/sdc1 2048 4095 2048 1M BIOS boot /dev/sdc2 4096 4198399 4194304 2G Linux filesystem /dev/sdc3 4198400 780138495 775940096 370G Linux filesystem Disk /dev/sdd: 1.45 TiB, 1599741100032 bytes, 3124494336 sectors Disk model: PERC H730 Mini Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/mapper/ubuntu--vg-ubuntu--lv: 370 GiB, 397280280576 bytes, 775938048 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes lvm output: root@ubuntu2204:~# vgscan Found volume group "ubuntu-vg" using metadata type lvm2 root@ubuntu2204:~# lvscan ACTIVE '/dev/ubuntu-vg/ubuntu-lv' [<370.00 GiB] inherit Before executing the command "rescan-scsi-bus.sh -r",the 'mount' output: root@ubuntu2204:~# mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=65903460k,nr_inodes=16475865,mode=755,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13191892k,mode=755,inode64) /dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=29226) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) /var/lib/snapd/snaps/core20_1494.snap on /snap/core20/1494 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/core20_1518.snap on /snap/core20/1518 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/lxd_22923.snap on /snap/lxd/22923 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/snapd_16010.snap on /snap/snapd/16010 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/snapd_16292.snap on /snap/snapd/16292 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /dev/sdc2 on /boot type ext4 (rw,relatime) tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13191892k,mode=755,inode64) nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=13191888k,nr_inodes=3297972,mode=700,inode64) during executing the command "rescan-scsi-bus.sh -r" output: root@ubuntu2204:~# rescan-scsi-bus.sh -r Syncing file systems Scanning SCSI subsystem for new devices and remove devices that have disappeared Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs sg4 changed: LU not available (PQual 3) REM: Host: scsi0 Channel: 02 Id: 02 Lun: 00 DEL: Vendor: DELL Model: PERC H730 Mini Rev: 4.30 Type: Direct-Access ANSI SCSI revision: 05 sg2 changed: LU not available (PQual 3) REM: Host: scsi0 Channel: 02 Id: 00 Lun: 00 DEL: Vendor: DELL Model: PERC H730 Mini Rev: 4.30 Type: Direct-Access ANSI SCSI revision: 05 sg3 changed: LU not available (PQual 3) REM: Host: scsi0 Channel: 02 Id: 01 Lun: 00 DEL: Vendor: DELL Model: PERC H730 Mini Rev: 4.30 Type: Direct-Access ANSI SCSI revision: 05 Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 10 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 11 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 12 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 13 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 14 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs sg5 changed: LU not available (PQual 3) REM: Host: scsi14 Channel: 00 Id: 00 Lun: 00 DEL: Vendor: HL-DT-ST Model: DVD-ROM DU90N Rev: D3C0 Type: CD-ROM ANSI SCSI revision: 05 Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 5 for all SCSI target IDs, all LUNs Scanning for device 5 0 0 0 ..sg0 changed: LU not available (PQual 3) REM: Host: scsi5 Channel: 00 Id: 00 Lun: 00 DEL: Vendor: EMC Model: LUNZ Rev: 7010 Type: Direct-Access ANSI SCSI revision: 04 Scanning host 6 for all SCSI target IDs, all LUNs Scanning for device 6 0 0 0 ..sg1 changed: LU not available (PQual 3) REM: Host: scsi6 Channel: 00 Id: 00 Lun: 00 DEL: Vendor: EMC Model: LUNZ Rev: 7010 Type: Direct-Access ANSI SCSI revision: 04 Scanning host 7 for all SCSI target IDs, all LUNs Scanning host 8 for all SCSI target IDs, all LUNs Scanning host 9 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs 0 new or changed device(s) found. 0 remapped or resized device(s) found. 6 device(s) removed. \t[0:2:2:0]\n\t[0:2:0:0]\n\t[0:2:1:0]\n\t[14:0:0:0]\n\t[5:0:0:0]\n\t[6:0:0:0]\nroot@ubuntu2204:~# After executing the command "rescan-scsi-bus.sh -r",the 'mount' output: root@ubuntu2204:~# mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=65903460k,nr_inodes=16475865,mode=755,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13191892k,mode=755,inode64) /dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (ro,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=29226) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) /var/lib/snapd/snaps/core20_1494.snap on /snap/core20/1494 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/core20_1518.snap on /snap/core20/1518 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/lxd_22923.snap on /snap/lxd/22923 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/snapd_16010.snap on /snap/snapd/16010 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/snapd_16292.snap on /snap/snapd/16292 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide) tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13191892k,mode=755,inode64) nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=13191888k,nr_inodes=3297972,mode=700,inode64) system log output during the "rescan-scsi-bus.sh -r": Jul 14 03:55:40 ubuntu2204 systemd[1]: Started Session 4 of User root. Jul 14 03:55:49 ubuntu2204 systemd[1]: session-4.scope: Deactivated successfully. Jul 14 03:55:52 ubuntu2204 systemd[1]: Stopping LVM event activation on device 8:35... Jul 14 03:55:52 ubuntu2204 systemd[1]: Stopping Unattended Upgrades Shutdown... Jul 14 03:55:52 ubuntu2204 systemd[1]: lvm2-pvscan@8:35.service: Deactivated successfully. Jul 14 03:55:52 ubuntu2204 systemd[1]: Stopped LVM event activation on device 8:35. Jul 14 03:55:52 ubuntu2204 systemd[1]: unattended-upgrades.service: Deactivated successfully. Jul 14 03:55:52 ubuntu2204 systemd[1]: Stopped Unattended Upgrades Shutdown. Jul 14 03:55:52 ubuntu2204 systemd[1]: Stopped target Local File Systems. Jul 14 03:55:52 ubuntu2204 systemd[1]: Unmounting /boot... Jul 14 03:55:52 ubuntu2204 kernel: [ 665.912071] Buffer I/O error on dev sdc2, logical block 262144, lost sync page write Jul 14 03:55:52 ubuntu2204 kernel: [ 665.912787] JBD2: Error -5 detected when updating journal superblock for sdc2-8. Jul 14 03:55:52 ubuntu2204 kernel: [ 665.913317] Aborting journal on device sdc2-8. Jul 14 03:55:52 ubuntu2204 kernel: [ 665.913636] Buffer I/O error on dev sdc2, logical block 262144, lost sync page write Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914189] JBD2: Error -5 detected when updating journal superblock for sdc2-8. Jul 14 03:55:52 ubuntu2204 systemd[1]: boot.mount: Deactivated successfully. Jul 14 03:55:52 ubuntu2204 systemd[1]: Unmounted /boot. Jul 14 03:55:52 ubuntu2204 systemd[1]: systemd-fsck@dev-disk-by\x2duuid-34aca05a\x2dd9ca\x2d46f4\x2d92db\x2d4d79251cb53c.service: Deactivated successfully. Jul 14 03:55:52 ubuntu2204 systemd[1]: Stopped File System Check on /dev/disk/by-uuid/34aca05a-d9ca-46f4-92db-4d79251cb53c. Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914551] EXT4-fs error (device sdc2): ext4_put_super:1189: comm umount: Couldn't clean up the journal Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914736] EXT4-fs (sdc2): Remounting filesystem read-only Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914946] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 84973083) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914953] Buffer I/O error on device dm-0, logical block 84973083 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914993] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28676) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.914996] Buffer I/O error on device dm-0, logical block 28676 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915022] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28678) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915024] Buffer I/O error on device dm-0, logical block 28678 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915050] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28689) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915052] Buffer I/O error on device dm-0, logical block 28689 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915079] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28780) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915080] Buffer I/O error on device dm-0, logical block 28780 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915106] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28788) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915107] Buffer I/O error on device dm-0, logical block 28788 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915133] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28823) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915134] Buffer I/O error on device dm-0, logical block 28823 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915159] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28851) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915161] Buffer I/O error on device dm-0, logical block 28851 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915188] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28853) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915190] Buffer I/O error on device dm-0, logical block 28853 Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915217] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 21236425 starting block 28903) Jul 14 03:55:52 ubuntu2204 kernel: [ 665.915218] Buffer I/O error on device dm-0, logical block 28903 Jul 14 03:55:52 ubuntu2204 kernel: [ 666.460639] ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 300) Jul 14 03:55:52 ubuntu2204 kernel: [ 666.464752] ata10.00: configured for UDMA/100 Jul 14 03:55:53 ubuntu2204 kernel: [ 667.686778] ata10.00: disabled Jul 14 03:55:57 ubuntu2204 kernel: [ 670.914630] Aborting journal on device dm-0-8. Jul 14 03:55:57 ubuntu2204 kernel: [ 670.914679] Buffer I/O error on dev dm-0, logical block 48267264, lost sync page write Jul 14 03:55:57 ubuntu2204 kernel: [ 670.914743] JBD2: Error -5 detected when updating journal superblock for dm-0-8. Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918526] EXT4-fs error (device dm-0): ext4_journal_check_start:83: comm rs:main Q:Reg: Detected aborted journal Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918558] EXT4-fs error (device dm-0): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918601] Buffer I/O error on dev dm-0, logical block 0, lost sync page write Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918703] EXT4-fs (dm-0): I/O error while writing superblock Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918710] EXT4-fs (dm-0): previous I/O error to superblock detected Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918726] EXT4-fs (dm-0): Remounting filesystem read-only Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918783] Buffer I/O error on dev dm-0, logical block 0, lost sync page write Jul 14 03:55:57 ubuntu2204 kernel: [ 670.918852] EXT4-fs (dm-0): I/O error while writing superblock Jul 14 03:56:25 ubuntu2204 multipathd[676]: sda: path already removed Jul 14 03:56:56 ubuntu2204 multipathd[676]: sdb: path already removed