| 2015-05-19 07:29:16 |
Tina Tang |
bug |
|
|
added bug |
| 2015-05-19 07:31:21 |
Tina Tang |
summary |
performance issue during volume detach ment when ISCSIConnector when iSCSI multipath |
Performance issue during volume detachment in ISCSIConnector when iSCSI multipath is used |
|
| 2015-05-19 07:32:48 |
Tina Tang |
description |
When the multipath is used, there is performance issue in the ISCSIConnector in the disconnect volume. When there are many attached volume, creating a volume from image may resulting in many of execution of 'multipath -ll <device>'
For example, on my system there are only 6 attached LUNs which leads to 35 iscsi devices files under /dev/disk/by-path:
stack@ubuntu-server12:/opt/stack/logs/screen$ ls /dev/disk/by-path | grep iscsi | wc -l
35
22492196-3071-4bb6-9dc2-41f0ce24269b is a request id of creating a volume from a image. There are 234 times of multipath -ll execution in order to to handle this request.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep 22492196-3071-4bb6-9dc2-41f0ce24269b screen-c-vol.log | grep "multipath \['\-ll'," | grep cinder.brick.initiator.connector | wc -l
231
When there are many attached LUNs on the host, there may be thousands of execution of "multipath -ll"
Per the debuging:
In the method _disconnect_volume_multipath_iscsi
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L488
Assume the number of iscsi devices under /dev/disk/by-path excludes the devices for the volume to be detached is N.
L494 - 501 will introduce N times of "multipath -ll <dev>"
L514 - 515 will introduce about N to N(N+1)/2 times of "multipath -ll <dev>" depends on the order of the devices.
Reproduce steps in Cinder:
1. Nova and Cinder is runing on a same host
2. Attach 50 luns to the VM created on the same host
3. Create a volume from image
This was found in Kilo release |
When the multipath is used, there is performance issue in the ISCSIConnector in the disconnect volume. When there are many attached volume, creating a volume from image may resulting in many of execution of 'multipath -ll <device>'
For example, on my system there are only 6 attached LUNs which leads to 35 iscsi devices files under /dev/disk/by-path:
stack@ubuntu-server12:/opt/stack/logs/screen$ ls /dev/disk/by-path | grep iscsi | wc -l
35
22492196-3071-4bb6-9dc2-41f0ce24269b is a request id of creating a volume from a image. There are 234 times of multipath -ll execution in order to to handle this request.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep 22492196-3071-4bb6-9dc2-41f0ce24269b screen-c-vol.log | grep "multipath \['\-ll'," | grep cinder.brick.initiator.connector | wc -l
231
When there are many attached LUNs on the host, there may be thousands of execution of "multipath -ll"
Per the debuging:
In the method _disconnect_volume_multipath_iscsi
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L488
Assume the number of iscsi devices under /dev/disk/by-path excludes the devices for the volume to be detached is N.
L494 - 501 will introduce N times of "multipath -ll <dev>"
L514 - 515 will introduce about N to N(N+1)/2 times of "multipath -ll <dev>" depends on the order of the devices.
Reproduce steps in Cinder:
1. Nova and Cinder is runing on a same host
2. Attach 50 luns to the VM created on the same host
3. Create a volume from image
This was found in Kilo release |
|
| 2015-05-19 08:10:39 |
Tina Tang |
description |
When the multipath is used, there is performance issue in the ISCSIConnector in the disconnect volume. When there are many attached volume, creating a volume from image may resulting in many of execution of 'multipath -ll <device>'
For example, on my system there are only 6 attached LUNs which leads to 35 iscsi devices files under /dev/disk/by-path:
stack@ubuntu-server12:/opt/stack/logs/screen$ ls /dev/disk/by-path | grep iscsi | wc -l
35
22492196-3071-4bb6-9dc2-41f0ce24269b is a request id of creating a volume from a image. There are 234 times of multipath -ll execution in order to to handle this request.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep 22492196-3071-4bb6-9dc2-41f0ce24269b screen-c-vol.log | grep "multipath \['\-ll'," | grep cinder.brick.initiator.connector | wc -l
231
When there are many attached LUNs on the host, there may be thousands of execution of "multipath -ll"
Per the debuging:
In the method _disconnect_volume_multipath_iscsi
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L488
Assume the number of iscsi devices under /dev/disk/by-path excludes the devices for the volume to be detached is N.
L494 - 501 will introduce N times of "multipath -ll <dev>"
L514 - 515 will introduce about N to N(N+1)/2 times of "multipath -ll <dev>" depends on the order of the devices.
Reproduce steps in Cinder:
1. Nova and Cinder is runing on a same host
2. Attach 50 luns to the VM created on the same host
3. Create a volume from image
This was found in Kilo release |
When the multipath is used, there is performance issue in the ISCSIConnector in the disconnect volume. When there are many attached volume, creating a volume from image may resulting in many of execution of 'multipath -ll <device>'
For example, on my system there are only 6 attached LUNs which leads to 35 iscsi devices files under /dev/disk/by-path:
stack@ubuntu-server12:/opt/stack/logs/screen$ ls /dev/disk/by-path | grep iscsi | wc -l
35
22492196-3071-4bb6-9dc2-41f0ce24269b is a request id of creating a volume from a image. There are 234 times of multipath -ll execution in order to to handle this request.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep 22492196-3071-4bb6-9dc2-41f0ce24269b screen-c-vol.log | grep "multipath \['\-ll'," | grep cinder.brick.initiator.connector | wc -l
231
When there are many attached LUNs on the host, there may be thousands of execution of "multipath -ll"
Per the debuging:
In the method _disconnect_volume_multipath_iscsi
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L488
Assume the number of iscsi devices under /dev/disk/by-path excludes the devices for the volume to be detached is N.
L494 - 501 will introduce N times of "multipath -ll <dev>"
L514 - 515 will introduce about N to N(N+1)/2 times of "multipath -ll <dev>" depends on the order of the devices.
When the driver is the default driver HostDriver
L494 - 501
It using multipath -ll to find the multipath dev path like "/dev/mapper/<multipath_id>" for each devices under /dev/disk/by-path
Then:
L514 -555
It go through each devices under /dev/disk/by-path to find the paths for each multipath dev found in L494 -501. And then parse the file name of the device under /dev/disk/by-path to get the used portal and iqns.
The whole logic is twist and has low performance. Why we need to find the multipath device first and then go back to find the paths under /dev/disk/by-path? Seems the logic can be simplified to go through the devices under /dev/disk/by-path to see whether the protals and iqns are used
Reproduce steps in Cinder:
1. Nova and Cinder is runing on a same host
2. Attach 50 luns to the VM created on the same host
3. Create a volume from image
This was found in Kilo release |
|
| 2015-05-19 08:11:27 |
Tina Tang |
description |
When the multipath is used, there is performance issue in the ISCSIConnector in the disconnect volume. When there are many attached volume, creating a volume from image may resulting in many of execution of 'multipath -ll <device>'
For example, on my system there are only 6 attached LUNs which leads to 35 iscsi devices files under /dev/disk/by-path:
stack@ubuntu-server12:/opt/stack/logs/screen$ ls /dev/disk/by-path | grep iscsi | wc -l
35
22492196-3071-4bb6-9dc2-41f0ce24269b is a request id of creating a volume from a image. There are 234 times of multipath -ll execution in order to to handle this request.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep 22492196-3071-4bb6-9dc2-41f0ce24269b screen-c-vol.log | grep "multipath \['\-ll'," | grep cinder.brick.initiator.connector | wc -l
231
When there are many attached LUNs on the host, there may be thousands of execution of "multipath -ll"
Per the debuging:
In the method _disconnect_volume_multipath_iscsi
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L488
Assume the number of iscsi devices under /dev/disk/by-path excludes the devices for the volume to be detached is N.
L494 - 501 will introduce N times of "multipath -ll <dev>"
L514 - 515 will introduce about N to N(N+1)/2 times of "multipath -ll <dev>" depends on the order of the devices.
When the driver is the default driver HostDriver
L494 - 501
It using multipath -ll to find the multipath dev path like "/dev/mapper/<multipath_id>" for each devices under /dev/disk/by-path
Then:
L514 -555
It go through each devices under /dev/disk/by-path to find the paths for each multipath dev found in L494 -501. And then parse the file name of the device under /dev/disk/by-path to get the used portal and iqns.
The whole logic is twist and has low performance. Why we need to find the multipath device first and then go back to find the paths under /dev/disk/by-path? Seems the logic can be simplified to go through the devices under /dev/disk/by-path to see whether the protals and iqns are used
Reproduce steps in Cinder:
1. Nova and Cinder is runing on a same host
2. Attach 50 luns to the VM created on the same host
3. Create a volume from image
This was found in Kilo release |
When the multipath is used, there is performance issue in the ISCSIConnector in the disconnect volume. When there are many attached volume, creating a volume from image may resulting in many of execution of 'multipath -ll <device>'
For example, on my system there are only 6 attached LUNs which leads to 35 iscsi devices files under /dev/disk/by-path:
stack@ubuntu-server12:/opt/stack/logs/screen$ ls /dev/disk/by-path | grep iscsi | wc -l
35
Given 22492196-3071-4bb6-9dc2-41f0ce24269b is a request id of creating a volume from a image. There are 231 times of multipath -ll execution in order to to handle this request.
stack@ubuntu-server12:/opt/stack/logs/screen$ grep 22492196-3071-4bb6-9dc2-41f0ce24269b screen-c-vol.log | grep "multipath \['\-ll'," | grep cinder.brick.initiator.connector | wc -l
231
When there are many attached LUNs on the host, there may be thousands of execution of "multipath -ll"
Per the debuging in to the code:
In the method _disconnect_volume_multipath_iscsi
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L488
Assume the number of iscsi devices under /dev/disk/by-path excludes the devices for the volume to be detached is N.
L494 - 501 will introduce N times of "multipath -ll <dev>"
L514 - 515 will introduce about N to N(N+1)/2 times of "multipath -ll <dev>" depends on the order of the devices.
When the driver is the default driver HostDriver
L494 - 501
It using multipath -ll to find the multipath dev path like "/dev/mapper/<multipath_id>" for each devices under /dev/disk/by-path
Then:
L514 -555
It go through each devices under /dev/disk/by-path to find the paths for each multipath dev found in L494 -501. And then parse the file name of the device under /dev/disk/by-path to get the used portal and iqns.
The whole logic is twist and has low performance. Why we need to find the multipath device first and then go back to find the paths under /dev/disk/by-path? Seems the logic can be simplified to go through the devices under /dev/disk/by-path to see whether the protals and iqns are used
Reproduce steps in Cinder:
1. Nova and Cinder is runing on a same host
2. Attach 50 luns to the VM created on the same host
3. Create a volume from image
This was found in Kilo release |
|
| 2015-05-22 18:44:51 |
Walt Boring |
os-brick: status |
New |
Triaged |
|
| 2015-05-22 18:57:32 |
rasoto |
os-brick: assignee |
|
rasoto (rasoto) |
|
| 2015-05-27 21:09:28 |
Walt Boring |
os-brick: importance |
Undecided |
Medium |
|
| 2015-06-11 23:48:03 |
Tomoki Sekiyama |
os-brick: assignee |
rasoto (rasoto) |
Tomoki Sekiyama (tsekiyama) |
|
| 2015-06-17 07:49:39 |
OpenStack Infra |
os-brick: status |
Triaged |
Fix Committed |
|
| 2015-07-20 15:28:14 |
Doug Hellmann |
os-brick: status |
Fix Committed |
Fix Released |
|
| 2015-07-20 15:28:14 |
Doug Hellmann |
os-brick: milestone |
|
0.3.0 |
|
| 2015-11-13 18:39:24 |
Preston L. Bannister |
bug |
|
|
added subscriber Preston L. Bannister |
| 2016-02-04 15:59:46 |
Eric Harney |
bug task added |
|
cinder |
|
| 2016-02-04 16:03:42 |
Eric Harney |
nominated for series |
|
cinder/kilo |
|
| 2016-02-04 16:07:27 |
Eric Harney |
bug task added |
|
cinder/kilo |
|
| 2016-02-04 16:07:36 |
Eric Harney |
cinder/kilo: milestone |
|
2015.1.3 |
|
| 2016-02-04 16:07:39 |
Eric Harney |
cinder/kilo: status |
New |
Fix Released |
|
| 2016-02-04 16:07:48 |
Eric Harney |
cinder: status |
New |
Invalid |
|
| 2016-02-04 19:15:02 |
Eric Harney |
cinder/kilo: assignee |
|
Patrick East (patrick-east) |
|
