Nova & Cinder - Disk attachement changes after shutoff/start

[Arpad Hajdu]

Description
===========
After shut off an instance and starting it again the disk path changes inside the virtual instance. This is usually a small issue if you are only using the disk label/or any other ID rather than disk path.
If any application is using the disk path inside windows the application becamomes failed until replacing the device path.

Also I understand that most people won't even notice the problem, but we don't have similar issues with VMware and we believe that the path should be the same regardless of any kind of shut off/startup operation.

Steps to reproduce
==================
- Deploy a new instance - eg: Windows 2012 R2
- Attach a new CEPH block device volume
- Verify the disk path inside Windows
    Device instance path: SCSI\DISK&VEN_RED_HAT&PROD_VIRTIO\4&1618751F&0&000000
 Parent: PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13c0b0c5&0&30
 Location: Bus Number 0, Target Id 0, LUN 0
- Shut off the instance
- Start the instance
- The disk is offline after the startup, changing to online working
- Verify the disk path:
 Device instance path: SCSI\DISK&VEN_RED_HAT&PROD_VIRTIO\4&352177D0&1&000000
 Parent: PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13c0b0c5&0&28
 Location: Bus Number 0, Target Id 0, LUN 0

Using Ubuntu 16.04 LTS:
- Spin up a new Instance using Ubuntu 16.04 LTS
- Attach a new CEPH block device
 root@attach-test-2:~# lsblk
 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
 vda 253:0 0 16G 0 disk
 └─vda1 253:1 0 16G 0 part /
 vdb 253:16 0 1G 0 disk

 root@attach-test-2:~# ls -la /dev/disk/by-path
 total 0
 drwxr-xr-x 2 root root 100 Jun 17 11:26 .
 drwxr-xr-x 6 root root 120 Jun 17 11:26 ..
 lrwxrwxrwx 1 root root 9 Jun 17 11:23 virtio-pci-0000:00:04.0 -> ../../vda
 lrwxrwxrwx 1 root root 10 Jun 17 11:23 virtio-pci-0000:00:04.0-part1 -> ../../vda1
 lrwxrwxrwx 1 root root 9 Jun 17 11:26 virtio-pci-0000:00:06.0 -> ../../vdb

- Shut ioff the instance
- Start the instance
- Check disk path
 root@attach-test-2:~# ls -la /dev/disk/by-path
 total 0
 drwxr-xr-x 2 root root 100 Jun 17 11:28 .
 drwxr-xr-x 6 root root 120 Jun 17 11:28 ..
 lrwxrwxrwx 1 root root 9 Jun 17 11:28 virtio-pci-0000:00:04.0 -> ../../vda
 lrwxrwxrwx 1 root root 10 Jun 17 11:28 virtio-pci-0000:00:04.0-part1 -> ../../vda1
 lrwxrwxrwx 1 root root 9 Jun 17 11:28 virtio-pci-0000:00:05.0 -> ../../vdb

THe disk path changed using Linux too

I have a feeling that this is a KVM/LibVirt issue...

Expected result
===============
Nova & Libvirt & KVM should use the same details when attaching the disk on the instance regardless how many time we shut it down and start it.
So both instance path and parent should be unchanged.

Actual result
=============
Path changes

Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii nova-api 2:13.0.0-0ubuntu2 all OpenStack Compute - API frontend
ii nova-cells 2:13.0.0-0ubuntu2 all Openstack Compute - cells
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7

Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7

Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7

Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7

Hypervisors:
KVM + Libvirt on all compute nodes

Storage:
CEPH, iSCSI and FC using Netapp

CEPH version - the same on all storage nodes
ii ceph 10.2.0-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.0-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph.
ii ceph-mds 10.2.0-0ubuntu0.16.04.1 amd64 metadata server for the ceph distributed file system
ii libcephfs1 10.2.0-0ubuntu0.16.04.1 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.0-0ubuntu0.16.04.1 amd64 Python libraries for the Ceph libcephfs library

Networking
Neutron networking with Openvswitch

----
root@openstack1:~# nova show Attach_Test_Win2k12

+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstackcompute3 |
| OS-EXT-SRV-ATTR:hostname | attach-test-win2k12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstackcompute3 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000114 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-jiddtkfq |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-17T10:44:43.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-06-17T10:44:27Z |
| description | Attach_Test_Win2k12 |
| flavor | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108) |
| hostId | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec |
| host_status | UP |
| id | 08c04ee9-b922-408d-8642-0e8f1fa59a1d |
| image | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068) |
| key_name | CCLTests |
| locked | False |
| metadata | {} |
| name | Attach_Test_Win2k12 |
| net-10.10.128 network | 10.10.128.51 |
| os-extended-volumes:volumes_attached | [{"id": "32595603-6c42-4bcb-818d-53b0c93b7c06", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | CCL |
| status | ACTIVE |
| tenant_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| updated | 2016-06-17T10:46:15Z |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+--------------------------------------+----------------------------------------------------------------------------------+
root@openstack1:~# openstack volume show 32595603-6c42-4bcb-818d-53b0c93b7c06
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'08c04ee9-b922-408d-8642-0e8f1fa59a1d', u'attachment_id': u'280a5b3b-599b-41a9-9793-ea6d774a1cb7', u'host_name': None, u'volume_id': u'32595603-6c42-4bcb-818d-53b0c93b7c06', u'device': u'/dev/vdb', u'id': u'32595603-6c42-4bcb-818d-53b0c93b7c06'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-06-17T10:53:38.000000 |
| description | |
| encrypted | False |
| id | 32595603-6c42-4bcb-818d-53b0c93b7c06 |
| migration_status | None |
| multiattach | False |
| name | Attachment_Test_1 |
| os-vol-host-attr:host | openstackblock.ccl-dev.net@rbd#CEPH |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | CEPH |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[Arpad Hajdu (hajduarpad-gmail)]

Libvirt.xml after the shut off attached

[PrashantUpadhyay (prashant-upadhyay)]

Hi Arpad,

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/persistent_naming.html

According to this documentation path based address in non persistent , it gets changed as soon as system is rebooted. So I think this should be expected behavior.

I think this naming is somewhat related to time taken while discovering the device. That is why it can not be persistent.

Please correct me if I am wrong.

~prashant

[Arpad Hajdu (hajduarpad-gmail)]

Hello,

I ran some other tests in our environment and I found the following:

- Attached the volume -> not reflected in the XML of the domain
- Shut down instance
- Start instance -> parent changed
- Shut down instance
- Start instance -> parent unchanged

To me it seems the issue is the KVM/Libvirt in this case. Actually this is a solution for us about how can we still use a Windows based application if it is using the full disk path... Attach a volume, shut down instance, start again and only start using the volume from this time.

[Sean Dague (sdague)]

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

[Javier Diaz Jr (javierdiazcharles)]

This is a very old and long running issue OpenStack has had with KVM. In essence, the KVM (libvirt) hypervisor lacks the ability to set the mount points on instances and thus regardless of what nova instructs the hypervisor to do KVM will simply attach the volume to the next available drive mapping (vdb, vdc, vdd) at boot.

This is unfortunately expected behavior if you ask me.