open-iscsi slow disk performance

Bug #305853 reported by Aaron Thul on 2008-12-07
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
open-iscsi (Ubuntu)
Undecided
Unassigned

Bug Description

Description: Ubuntu 8.04.1
Release: 8.04
open-iscsi:
  Installed: 2.0.865-1ubuntu3
  Candidate: 2.0.865-1ubuntu3
  Version table:
 *** 2.0.865-1ubuntu3 0
        500 http://us.archive.ubuntu.com hardy/main Packages
        100 /var/lib/dpkg/status

When the server boots up from a complete power down the IO speed of the iscsi disks performs 100 times slower than it should. To resolve this problem, after weeks of trying different things, I force downgrade to a older open-iscsi package and then upgrade right back using this command:
dpkg -i /root/open-iscsi_2.0.730-1_amd64.deb ; /etc/init.d/open-iscsi restart ; aptitude install open-iscsi
Every time I perform this step the IO performance immediately jumps back up to where it is suppose to be.

I am having this problem on two Ubuntu 8.04.1 servers one running Dell Hardware the other Supermicro. Nothing else in the data center is having any IO performance problems with the Iscsi device.

Aaron Thul (aaron-thul) wrote :
Download full text (3.1 KiB)

I have been doing some experimenting and I found something I do not understand and it may have something to do with this problem. I recently took node that when the iscsi disks are running slow I am missing a process, specifically: /usr/sbin/iscsid.

It starts out that iscsi is running slow, I take a look and see what iscsi is doing...
root@mara:~# ps -ef | grep scsi
root 1493 2 0 2008 ? 00:00:00 [scsi_eh_0]
root 2433 2 0 2008 ? 00:00:00 [scsi_eh_1]
root 2434 2 0 2008 ? 00:00:00 [scsi_eh_2]
root 5138 2 0 2008 ? 00:00:00 [scsi_eh_3]
root 5139 2 0 2008 ? 00:23:42 [scsi_wq_3]
root 26621 26468 0 12:13 pts/0 00:00:00 grep scsi
root 27919 1 0 Jan05 ? 00:00:00 /sbin/iscsid

I do my usually parlor trick to get it running at proper speed again:
root@mara:~# dpkg -i /home/apthul/open-iscsi_2.0.730-1_amd64.deb ; aptitude install open-iscsi
dpkg - warning: downgrading open-iscsi from 2.0.865-1ubuntu3 to 2.0.730-1.
(Reading database ... 27321 files and directories currently installed.)
Preparing to replace open-iscsi 2.0.865-1ubuntu3 (using .../open-iscsi_2.0.730-1_amd64.deb) ...
 * Disconnecting iSCSI targets
   ...done.
 * Stopping iSCSI initiator service
   ...done.
Unpacking replacement open-iscsi ...
Setting up open-iscsi (2.0.730-1) ...
Installing new version of config file /etc/init.d/open-iscsi ...
 * Starting iSCSI initiator service iscsid
   ...done.
 * Setting up iSCSI targets
   ...done.

Preparing to replace open-iscsi 2.0.730-1 (using .../open-iscsi_2.0.865-1ubuntu3_amd64.deb) ...
 * Disconnecting iSCSI targets
   ...done.
 * Stopping iSCSI initiator service
   ...done.
Unpacking replacement open-iscsi ...
Setting up open-iscsi (2.0.865-1ubuntu3) ...
Installing new version of config file /etc/init.d/open-iscsi ...
update-rc.d: /etc/init.d/remove: file does not exist
 * Starting iSCSI initiator service iscsid
   ...done.
 * Setting up iSCSI targets

Everything test out well and I decide I wanted to see what was running now:
apthul@mara:~$ ps -ef | grep scsi
root 1493 2 0 2008 ? 00:00:00 [scsi_eh_0]
root 2433 2 0 2008 ? 00:00:00 [scsi_eh_1]
root 2434 2 0 2008 ? 00:00:00 [scsi_eh_2]
root 5138 2 0 2008 ? 00:00:00 [scsi_eh_3]
root 5139 2 0 2008 ? 00:23:42 [scsi_wq_3]
root 26742 1 0 12:13 ? 00:00:00 /usr/sbin/iscsid ##What the heck is that!???
root 26960 1 0 12:13 ? 00:00:00 /sbin/iscsid
root 26961 1 0 12:13 ? 00:00:00 /sbin/iscsid
apthul 27018 22441 0 12:16 pts/0 00:00:00 grep scsi
apthul@mara:~$ iostat -cdmtx 10 999999

I decided to go see what it was:
apthul@mara:~$ ls -latr /usr/sbin/iscsid
ls: cannot access /usr/sbin/iscsid: No such file or directory

It is not there but /usr/sbin/iscsid is still running, it must have been deleted from the open-scsi upgrade. Could this be the magic that is keeping my open-iscsi running at proper speeds? Could it be that /usr/sbin/iscsid running is causing libraries to not be reloaded that are keeping the open-iscsi running prope...

Read more...

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers