| 2016-09-14 22:52:47 |
Patrick East |
bug |
|
|
added bug |
| 2016-09-20 14:42:35 |
Gustavo Randich |
bug |
|
|
added subscriber Gustavo Randich |
| 2016-11-04 21:55:43 |
OpenStack Infra |
os-brick: status |
New |
Fix Released |
|
| 2017-01-25 12:18:02 |
Hua Zhang |
summary |
multipath iscsi does not logout of sessions on xenial |
[SRU] multipath iscsi does not logout of sessions on xenial |
|
| 2017-01-25 12:19:58 |
Hua Zhang |
description |
Using ubuntu 16.04 devstack w/ Pure iSCSI driver and multipathing when detaching a volume the iscsi sessions are not logged out. This leaves behind a mpath device and the iscsi /dev/disk/by-path devices as broken luns
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-attach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d1d68e04-a217-44ea-bb74-65e0de73e5f8 |
| serverId | 6e1017a7-6dea-418f-ad9b-879da085bd13 |
| volumeId | d1d68e04-a217-44ea-bb74-65e0de73e5f8 |
+----------+--------------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| d1d68e04-a217-44ea-bb74-65e0de73e5f8 | in-use | - | 1 | pure-iscsi | false | 6e1017a7-6dea-418f-ad9b-879da085bd13 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ nova list
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
| 6e1017a7-6dea-418f-ad9b-879da085bd13 | test | ACTIVE | - | Running | public=172.24.4.12, 2001:db8::b |
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m node
10.0.1.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.1.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get udev uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sysfs uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sgio uid: No such file or directory
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.163521] audit: type=1400 audit(1473892394.556:21): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.173614] audit: type=1400 audit(1473892394.568:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13//qemu_bridge_helper" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester iscsid: Connection8:0 to [target: iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873, portal: 10.0.5.11,3260] through [iface: default] is operational now
^C
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-detach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d1d68e04-a217-44ea-bb74-65e0de73e5f8 | available | - | 1 | pure-iscsi | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@xenial-devstack-master-master-20160914-092014:~$ iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:48:10 xenial-qemu-tester kernel: [23257.736455] connection6:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742036] connection5:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742066] connection7:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742139] connection8:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742156] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747638] connection5:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747666] connection7:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747710] connection8:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747737] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester iscsid: message repeated 67 times: [ conn 0 login rejected: initiator failed authorization with target]
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.753999] connection6:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754019] connection8:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754105] connection5:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754146] connection7:0: detected conn error (1020) |
[Impact]
* The reload (multipath -r) in _rescan_multipath can cause /dev/mapper/<wwid> to be deleted and re-created (bug #1621340 is used to track this problem), it would cause a lot more downstream openstack issues. For example, and right in between that, os.stat(mdev) called by _discover_mpath_device() will fail to find the file. For example, when detaching a volume the iscsi sessions are not logged out. This leaves behind a mpath device and the iscsi /dev/disk/by-path devices as broken luns. So we should stop calling multipath -r when attaching/detaching iSCSI volumes, multipath will load devices on its own.
[Test Case]
* Enable iSCSI driver and cinder/nova multipath
* Detach a iSCSI volume
* Check that devices/symlinks do not get messed up mentioned below
[Regression Potential]
* None
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-attach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d1d68e04-a217-44ea-bb74-65e0de73e5f8 |
| serverId | 6e1017a7-6dea-418f-ad9b-879da085bd13 |
| volumeId | d1d68e04-a217-44ea-bb74-65e0de73e5f8 |
+----------+--------------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| d1d68e04-a217-44ea-bb74-65e0de73e5f8 | in-use | - | 1 | pure-iscsi | false | 6e1017a7-6dea-418f-ad9b-879da085bd13 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ nova list
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
| 6e1017a7-6dea-418f-ad9b-879da085bd13 | test | ACTIVE | - | Running | public=172.24.4.12, 2001:db8::b |
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m node
10.0.1.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.1.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get udev uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sysfs uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sgio uid: No such file or directory
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.163521] audit: type=1400 audit(1473892394.556:21): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.173614] audit: type=1400 audit(1473892394.568:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13//qemu_bridge_helper" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester iscsid: Connection8:0 to [target: iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873, portal: 10.0.5.11,3260] through [iface: default] is operational now
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-detach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d1d68e04-a217-44ea-bb74-65e0de73e5f8 | available | - | 1 | pure-iscsi | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@xenial-devstack-master-master-20160914-092014:~$ iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:48:10 xenial-qemu-tester kernel: [23257.736455] connection6:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742036] connection5:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742066] connection7:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742139] connection8:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742156] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747638] connection5:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747666] connection7:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747710] connection8:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747737] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester iscsid: message repeated 67 times: [ conn 0 login rejected: initiator failed authorization with target]
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.753999] connection6:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754019] connection8:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754105] connection5:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754146] connection7:0: detected conn error (1020) |
|
| 2017-01-25 12:20:53 |
Hua Zhang |
tags |
|
sts-sru |
|
| 2017-01-25 12:22:03 |
Hua Zhang |
attachment added |
|
fix-1623700-xenial.debdiff https://bugs.launchpad.net/os-brick/+bug/1623700/+attachment/4808919/+files/fix-1623700-xenial.debdiff |
|
| 2017-01-25 12:26:26 |
Hua Zhang |
bug task added |
|
python-os-brick (Ubuntu) |
|
| 2017-01-25 12:26:48 |
Hua Zhang |
python-os-brick (Ubuntu): status |
New |
Invalid |
|
| 2017-01-25 12:28:09 |
Hua Zhang |
bug task deleted |
python-os-brick (Ubuntu) |
|
|
| 2017-01-26 08:20:30 |
Louis Bouchard |
bug task added |
|
python-os-brick (Ubuntu) |
|
| 2017-01-26 08:20:39 |
Louis Bouchard |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-01-26 08:20:39 |
Louis Bouchard |
bug task added |
|
python-os-brick (Ubuntu Xenial) |
|
| 2017-01-26 08:26:46 |
Hua Zhang |
python-os-brick (Ubuntu): status |
New |
Invalid |
|
| 2017-01-26 08:26:52 |
Hua Zhang |
python-os-brick (Ubuntu Xenial): assignee |
|
Hua Zhang (zhhuabj) |
|
| 2017-01-26 09:22:39 |
Edward Hope-Morley |
bug task added |
|
cloud-archive |
|
| 2017-01-26 09:25:01 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/mitaka |
|
| 2017-02-06 11:16:16 |
Edward Hope-Morley |
python-os-brick (Ubuntu): status |
Invalid |
Fix Released |
|
| 2017-02-06 11:16:18 |
Edward Hope-Morley |
python-os-brick (Ubuntu Xenial): status |
New |
In Progress |
|
| 2017-02-13 03:55:28 |
Hua Zhang |
bug |
|
|
added subscriber Ubuntu Server Team |
| 2017-02-13 03:58:09 |
Hua Zhang |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2017-02-14 04:43:25 |
Mathew Hodson |
python-os-brick (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-02-14 04:43:27 |
Mathew Hodson |
python-os-brick (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2017-02-23 01:35:19 |
Billy Olsen |
tags |
sts-sru |
sts sts-sponsor sts-sru ubuntu-sponsors |
|
| 2017-03-17 11:27:58 |
Hua Zhang |
attachment added |
|
fix-1623700-xenial-v2.debdiff https://bugs.launchpad.net/os-brick/+bug/1623700/+attachment/4839388/+files/fix-1623700-xenial-v2.debdiff |
|
| 2017-03-17 14:48:40 |
Corey Bryant |
bug task added |
|
cloud-archive/mitaka |
|
| 2017-03-17 14:49:00 |
Corey Bryant |
nominated for series |
|
cloud-archive/newton |
|
| 2017-03-17 14:49:00 |
Corey Bryant |
bug task added |
|
cloud-archive/newton |
|
| 2017-03-17 14:49:07 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2017-03-17 14:49:10 |
Corey Bryant |
cloud-archive: importance |
Undecided |
Medium |
|
| 2017-03-17 14:49:11 |
Corey Bryant |
cloud-archive/mitaka: importance |
Undecided |
Medium |
|
| 2017-03-17 14:49:13 |
Corey Bryant |
cloud-archive/newton: importance |
Undecided |
Medium |
|
| 2017-03-17 14:49:20 |
Corey Bryant |
cloud-archive/newton: status |
New |
Triaged |
|
| 2017-03-17 14:49:22 |
Corey Bryant |
cloud-archive/mitaka: status |
New |
Triaged |
|
| 2017-03-17 16:14:57 |
Louis Bouchard |
nominated for series |
|
Ubuntu Yakkety |
|
| 2017-03-17 16:14:57 |
Louis Bouchard |
bug task added |
|
python-os-brick (Ubuntu Yakkety) |
|
| 2017-03-17 16:15:11 |
Louis Bouchard |
python-os-brick (Ubuntu Yakkety): importance |
Undecided |
Medium |
|
| 2017-03-17 16:15:18 |
Louis Bouchard |
python-os-brick (Ubuntu Yakkety): status |
New |
Triaged |
|
| 2017-03-22 12:48:47 |
Hua Zhang |
description |
[Impact]
* The reload (multipath -r) in _rescan_multipath can cause /dev/mapper/<wwid> to be deleted and re-created (bug #1621340 is used to track this problem), it would cause a lot more downstream openstack issues. For example, and right in between that, os.stat(mdev) called by _discover_mpath_device() will fail to find the file. For example, when detaching a volume the iscsi sessions are not logged out. This leaves behind a mpath device and the iscsi /dev/disk/by-path devices as broken luns. So we should stop calling multipath -r when attaching/detaching iSCSI volumes, multipath will load devices on its own.
[Test Case]
* Enable iSCSI driver and cinder/nova multipath
* Detach a iSCSI volume
* Check that devices/symlinks do not get messed up mentioned below
[Regression Potential]
* None
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-attach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d1d68e04-a217-44ea-bb74-65e0de73e5f8 |
| serverId | 6e1017a7-6dea-418f-ad9b-879da085bd13 |
| volumeId | d1d68e04-a217-44ea-bb74-65e0de73e5f8 |
+----------+--------------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| d1d68e04-a217-44ea-bb74-65e0de73e5f8 | in-use | - | 1 | pure-iscsi | false | 6e1017a7-6dea-418f-ad9b-879da085bd13 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ nova list
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
| 6e1017a7-6dea-418f-ad9b-879da085bd13 | test | ACTIVE | - | Running | public=172.24.4.12, 2001:db8::b |
+--------------------------------------+------+--------+------------+-------------+---------------------------------+
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m node
10.0.1.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.1.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get udev uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sysfs uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sgio uid: No such file or directory
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.163521] audit: type=1400 audit(1473892394.556:21): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.173614] audit: type=1400 audit(1473892394.568:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13//qemu_bridge_helper" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester iscsid: Connection8:0 to [target: iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873, portal: 10.0.5.11,3260] through [iface: default] is operational now
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-detach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d1d68e04-a217-44ea-bb74-65e0de73e5f8 | available | - | 1 | pure-iscsi | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@xenial-devstack-master-master-20160914-092014:~$ iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:48:10 xenial-qemu-tester kernel: [23257.736455] connection6:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742036] connection5:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742066] connection7:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742139] connection8:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742156] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747638] connection5:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747666] connection7:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747710] connection8:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747737] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester iscsid: message repeated 67 times: [ conn 0 login rejected: initiator failed authorization with target]
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.753999] connection6:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754019] connection8:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754105] connection5:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754146] connection7:0: detected conn error (1020) |
[Impact]
* multipath-tools has a bug that 'multipath -r' can cause /dev/mapper/<wwid> to be deleted and re-created momentarily (bug #1621340 is used to track this problem), thus os.stat(mdev) right after _rescan_multipath ('multipath -r') in os-brick can fail if it is executed before multipath dev being re-created. This will also lead to multipath iscsi does not logout of sessions on xenial.
[Test Case]
* Enable cinder multipath by adding iscsi_ip_address and iscsi_secondary_ip_addresses in cinder.conf
* Enable nova multipath by adding iscsi_use_multipath=True in [libvirt] secion of nova.conf
* Detach a iSCSI volume
* Check that devices/symlinks do not get messed up mentioned below, or check that multipath device /dev/mapper/<wwid> doesn't be deleted and re-created momentarily
[Regression Potential]
* multipath-tools loads devices on its own, we shouldn't need to be forcing multipathd to do reload, so there is no regression potential.
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m node
10.0.1.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.11:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.5.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
10.0.1.10:3260,-1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get udev uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sysfs uid: Invalid argument
Sep 14 22:33:14 xenial-qemu-tester multipath: dm-0: failed to get sgio uid: No such file or directory
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1347]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-scsi\x2d3624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester systemd[1]: dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device: Dev dev-disk-by\x2did-wwn\x2d0x624a93709a738ed78583fd12003fb774.device appeared twice with different sysfs paths /sys/devices/platform/host6/session5/target6:0:0/6:0:0:1/block/sda and /sys/devices/virtual/block/dm-0
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.163521] audit: type=1400 audit(1473892394.556:21): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester kernel: [22362.173614] audit: type=1400 audit(1473892394.568:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-6e1017a7-6dea-418f-ad9b-879da085bd13//qemu_bridge_helper" pid=32665 comm="apparmor_parser"
Sep 14 22:33:14 xenial-qemu-tester iscsid: Connection8:0 to [target: iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873, portal: 10.0.5.11,3260] through [iface: default] is operational now
stack@xenial-devstack-master-master-20160914-092014:~$ nova volume-detach 6e1017a7-6dea-418f-ad9b-879da085bd13 d1d68e04-a217-44ea-bb74-65e0de73e5f8
stack@xenial-devstack-master-master-20160914-092014:~$ sudo iscsiadm -m session
tcp: [5] 10.0.1.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [6] 10.0.5.10:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [7] 10.0.1.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
tcp: [8] 10.0.5.11:3260,1 iqn.2010-06.com.purestorage:flasharray.3adbe40b49bac873 (non-flash)
stack@xenial-devstack-master-master-20160914-092014:~$ sudo tail -f /var/log/syslog
Sep 14 22:48:10 xenial-qemu-tester kernel: [23257.736455] connection6:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742036] connection5:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742066] connection7:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742139] connection8:0: detected conn error (1020)
Sep 14 22:48:13 xenial-qemu-tester kernel: [23260.742156] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747638] connection5:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747666] connection7:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747710] connection8:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester kernel: [23263.747737] connection6:0: detected conn error (1020)
Sep 14 22:48:16 xenial-qemu-tester iscsid: message repeated 67 times: [ conn 0 login rejected: initiator failed authorization with target]
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.753999] connection6:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754019] connection8:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754105] connection5:0: detected conn error (1020)
Sep 14 22:48:19 xenial-qemu-tester kernel: [23266.754146] connection7:0: detected conn error (1020) |
|
| 2017-03-22 12:49:42 |
Hua Zhang |
attachment added |
|
fix-1623700-yaketty.debdiff https://bugs.launchpad.net/os-brick/+bug/1623700/+attachment/4842276/+files/fix-1623700-yaketty.debdiff |
|
| 2017-03-23 11:28:12 |
Edward Hope-Morley |
tags |
sts sts-sponsor sts-sru ubuntu-sponsors |
sts sts-sponsor sts-sru-needed |
|
| 2017-03-29 20:05:19 |
Michael Terry |
bug |
|
|
added subscriber Michael Terry |
| 2017-03-29 20:05:22 |
Michael Terry |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2017-03-29 20:05:27 |
Michael Terry |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-04-03 09:18:39 |
Edward Hope-Morley |
tags |
sts sts-sponsor sts-sru-needed |
sts sts-sru-needed |
|
| 2017-04-05 13:19:31 |
Chris J Arges |
python-os-brick (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2017-04-05 13:19:35 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2017-04-05 13:19:41 |
Chris J Arges |
tags |
sts sts-sru-needed |
sts sts-sru-needed verification-needed |
|
| 2017-04-05 13:20:21 |
Chris J Arges |
python-os-brick (Ubuntu Yakkety): status |
Triaged |
Fix Committed |
|
| 2017-04-12 06:56:34 |
Hua Zhang |
tags |
sts sts-sru-needed verification-needed |
sts sts-sru-needed verification-xenial-done verification-yakkety-done |
|
| 2017-04-24 11:22:49 |
James Page |
tags |
sts sts-sru-needed verification-xenial-done verification-yakkety-done |
sts sts-sru-needed verification-done |
|
| 2017-04-24 11:26:34 |
James Page |
cloud-archive/mitaka: status |
Triaged |
Fix Committed |
|
| 2017-04-24 11:26:35 |
James Page |
tags |
sts sts-sru-needed verification-done |
sts sts-sru-needed verification-done verification-mitaka-needed |
|
| 2017-04-24 11:27:35 |
James Page |
cloud-archive/newton: status |
Triaged |
Fix Committed |
|
| 2017-04-24 11:27:36 |
James Page |
tags |
sts sts-sru-needed verification-done verification-mitaka-needed |
sts sts-sru-needed verification-done verification-mitaka-needed verification-newton-needed |
|
| 2017-04-25 12:29:41 |
Launchpad Janitor |
python-os-brick (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-04-25 12:29:46 |
Robie Basak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-04-25 12:29:54 |
Launchpad Janitor |
python-os-brick (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-04-26 07:08:09 |
Hua Zhang |
tags |
sts sts-sru-needed verification-done verification-mitaka-needed verification-newton-needed |
sts sts-sru-needed verification-done verification-mitaka-done verification-newton-done |
|
| 2017-05-15 08:31:35 |
James Page |
cloud-archive/mitaka: status |
Fix Committed |
Fix Released |
|
| 2017-05-15 08:34:48 |
James Page |
cloud-archive/newton: status |
Fix Committed |
Fix Released |
|
