I do not see any difference between package 2011.3-0ubuntu6 and 2011.3-0ubuntu6.1 (oneiric-proposed). For me, it looks like the problem is still there: nova-compute is still looking for lun-0 when it should look for lun-1. lun-0 is reserved by tgt for the controler. Bellow the three cases I have tested: A) instance on node2 and volume on node1: NOK B) instance and volume on node2: OK (but iSCSI not used in such case) C) instance on node1 and volume on node2: NOK Case A) and C) give the same error. --- node1: 172.17.68.16 (nova-api, nova-objectstore, nova-scheduler, nova-network, mysql, rabbitmq, glance, nova-compute, nova-volume) node2: 172.17.68.100 (nova-compute, nova-volume) root@node1:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=11.10 DISTRIB_CODENAME=oneiric DISTRIB_DESCRIPTION="Ubuntu 11.10" root@node2:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=11.10 DISTRIB_CODENAME=oneiric DISTRIB_DESCRIPTION="Ubuntu 11.10" I have upgraded all the packages from oneiric-proposed (not only nova, but also open-iscsi, etc.): node1: 24 packages upgraded: [UPGRADE] initramfs-tools 0.99ubuntu7 -> 0.99ubuntu8 [UPGRADE] initramfs-tools-bin 0.99ubuntu7 -> 0.99ubuntu8 [UPGRADE] libgssapi-krb5-2 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libk5crypto3 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libkrb5-3 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libkrb5support0 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libpam-modules 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libpam-modules-bin 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libpam-runtime 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libpam0g 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libxenstore3.0 4.1.1-2ubuntu4 -> 4.1.1-2ubuntu4.1 [UPGRADE] nova-api 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-common 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-compute 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-compute-kvm 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-doc 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-network 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-objectstore 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-scheduler 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-volume 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] open-iscsi 2.0.871-0ubuntu8 -> 2.0.871-0ubuntu9 [UPGRADE] open-iscsi-utils 2.0.871-0ubuntu8 -> 2.0.871-0ubuntu9 [UPGRADE] python-nova 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] update-manager-core 1:0.152.25 -> 1:0.152.25.2 node2: 20 packages upgraded: [UPGRADE] initramfs-tools 0.99ubuntu7 -> 0.99ubuntu8 [UPGRADE] initramfs-tools-bin 0.99ubuntu7 -> 0.99ubuntu8 [UPGRADE] libgssapi-krb5-2 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libk5crypto3 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libkrb5-3 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libkrb5support0 1.9.1+dfsg-1ubuntu1 -> 1.9.1+dfsg-1ubuntu2 [UPGRADE] libpam-modules 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libpam-modules-bin 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libpam-runtime 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libpam0g 1.1.3-2ubuntu1 -> 1.1.3-2ubuntu2 [UPGRADE] libxenstore3.0 4.1.1-2ubuntu4 -> 4.1.1-2ubuntu4.1 [UPGRADE] nova-common 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-compute 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-compute-kvm 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-doc 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] nova-volume 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] open-iscsi 2.0.871-0ubuntu8 -> 2.0.871-0ubuntu9 [UPGRADE] open-iscsi-utils 2.0.871-0ubuntu8 -> 2.0.871-0ubuntu9 [UPGRADE] python-nova 2011.3-0ubuntu6 -> 2011.3-0ubuntu6.1 [UPGRADE] update-manager-core 1:0.152.25 -> 1:0.152.25.2 root@node1:~# nova-manage service list Binary Host Zone Status State Updated_At nova-scheduler node1 nova enabled :-) 2011-10-18 12:06:45 nova-network node1 nova enabled :-) 2011-10-18 12:06:41 nova-compute node1 nova enabled :-) 2011-10-18 12:06:46 nova-volume node1 nova enabled :-) 2011-10-18 12:06:42 nova-volume node2 nova enabled :-) 2011-10-18 12:06:42 nova-compute node2 nova enabled :-) 2011-10-18 12:06:43 root@node1:~# dpkg -l | grep nova ii nova-api 2011.3-0ubuntu6.1 OpenStack Compute - API frontend ii nova-common 2011.3-0ubuntu6.1 OpenStack Compute - common files ii nova-compute 2011.3-0ubuntu6.1 OpenStack Compute - compute node ii nova-compute-kvm 2011.3-0ubuntu6.1 OpenStack Compute - compute node (KVM) ii nova-doc 2011.3-0ubuntu6.1 OpenStack Compute - documetation ii nova-network 2011.3-0ubuntu6.1 OpenStack Compute - Network manager ii nova-objectstore 2011.3-0ubuntu6.1 OpenStack Compute - object store ii nova-scheduler 2011.3-0ubuntu6.1 OpenStack Compute - virtual machine scheduler ii nova-volume 2011.3-0ubuntu6.1 OpenStack Compute - storage ii python-nova 2011.3-0ubuntu6.1 OpenStack Compute Python libraries ii python-novaclient 2.6.4~bzr112-0ubuntu1 client library for OpenStack Compute API root@node2:~# dpkg -l | grep nova ii nova-common 2011.3-0ubuntu6.1 OpenStack Compute - common files ii nova-compute 2011.3-0ubuntu6.1 OpenStack Compute - compute node ii nova-compute-kvm 2011.3-0ubuntu6.1 OpenStack Compute - compute node (KVM) ii nova-doc 2011.3-0ubuntu6.1 OpenStack Compute - documetation ii nova-volume 2011.3-0ubuntu6.1 OpenStack Compute - storage ii python-nova 2011.3-0ubuntu6.1 OpenStack Compute Python libraries ii python-novaclient 2.6.4~bzr112-0ubuntu1 client library for OpenStack Compute API ******************************************************************************************** A) instance on node2 and volume on node1: NOK ******************************************************************************************** root@client:~# euca-create-volume -s 1 -z nova VOLUME vol-00000008 1 creating (project-one, None, None, None) 2011-10-18T12:01:13Z root@client:~# euca-describe-volumes VOLUME vol-00000008 1 nova available (project-one, node1, None, None) 2011-10-18T12:01:13Z root@node1:~# lvdisplay --- Logical volume --- LV Name /dev/nova-volumes/volume-00000008 VG Name nova-volumes LV UUID jH5OkI-idHM-WJ4b-8lez-o2N7-L1RX-zIAT7N LV Write Access read/write LV Status available # open 1 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:2 root@node1:~# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2010-10.org.openstack:volume-00000008 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 1074 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/nova-volumes/volume-00000008 Backing store flags: Account information: ACL information: ALL root@client:~# euca-describe-images IMAGE ami-00000002 None (oneiric-server-cloudimg-amd64) available public x86_64 machine aki-00000001 instance-store IMAGE aki-00000001 None (oneiric-server-cloudimg-amd64-kernel) available public x86_64 kernel instance-store root@client:~# euca-run-instances -k key-dodeeric -t m1.medium ami-00000002 RESERVATION r-sr4ftljp project-one default INSTANCE i-00000004 ami-00000002 pending key-dodeeric (project-one, None) 0 m1.medium 2011-10-18T12:09:42Z unknown zone aki-00000001 ami-00000000 root@client:~# euca-describe-instances RESERVATION r-sr4ftljp project-one default INSTANCE i-00000004 ami-00000002 10.0.1.3 10.0.1.3 running key-dodeeric (project-one, node2) 0 m1.medium 2011-10-18T12:09:42Z nova aki-00000001 ami-00000000 root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdd vol-00000008 VOLUME vol-00000008 root@node1:~# euca-describe-volumes VOLUME vol-00000008 1 nova available (project-one, node1, None, None) 2011-10-18T12:01:13Z ==========> NOK (not attached) root@node2:~# tail -200 /var/log/nova/nova-compute.log 2011-10-18 14:11:39,174 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'92867742-133c-4676-baa3-27b99f013277', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdd', u'volume_id': 8}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-18T12:11:39.081979', u'_context_user_id': u'dodeeric', u'method': u'attach_volume', u'_context_remote_address': u'172.17.68.16'} from (pid=1013) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2011-10-18 14:11:39,175 DEBUG nova.rpc [-] unpacked context: {'user_id': u'dodeeric', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-18T12:11:39.081979', 'auth_token': None, 'msg_id': None, 'remote_address': u'172.17.68.16', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'92867742-133c-4676-baa3-27b99f013277', 'project_id': u'project-one', 'read_deleted': False} from (pid=1013) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2011-10-18 14:11:39,176 INFO nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] check_instance_lock: decorating: || 2011-10-18 14:11:39,177 INFO nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] check_instance_lock: arguments: || || |4| 2011-10-18 14:11:39,177 DEBUG nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] instance 4: getting locked state from (pid=1013) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165 2011-10-18 14:11:39,250 INFO nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] check_instance_lock: locked: |False| 2011-10-18 14:11:39,251 INFO nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] check_instance_lock: admin: |True| 2011-10-18 14:11:39,251 INFO nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] check_instance_lock: executing: || 2011-10-18 14:11:39,324 AUDIT nova.compute.manager [92867742-133c-4676-baa3-27b99f013277 dodeeric project-one] instance 4: attaching volume 8 to /dev/vdd 2011-10-18 14:11:39,362 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery 2011-10-18 14:11:39,363 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1013) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:11:39,445 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 172.17.68.16:3260,1 iqn.2010-10.org.openstack:volume-00000008 from (pid=1013) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479 2011-10-18 14:11:39,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000008 -p 172.17.68.16:3260 --login from (pid=1013) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:11:40,011 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000008, portal: 172.17.68.16,3260] Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000008, portal: 172.17.68.16,3260]: successful stderr= from (pid=1013) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:11:40,017 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000008 -p 172.17.68.16:3260 --op update -n node.startup -v automatic from (pid=1013) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:11:40,087 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1013) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:11:40,091 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-172.17.68.16:3260-iscsi-iqn.2010-10.org.openstack:volume-00000008-lun-0. Will rescan & retry. Try number: 0 2011-10-18 14:11:40,092 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000008 -p 172.17.68.16:3260 --rescan from (pid=1013) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:11:40,149 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 2, target: iqn.2010-10.org.openstack:volume-00000008, portal: 172.17.68.16,3260] stderr= from (pid=1013) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:11:41,153 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-172.17.68.16:3260-iscsi-iqn.2010-10.org.openstack:volume-00000008-lun-0. Will rescan & retry. Try number: 1 2011-10-18 14:11:41,154 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000008 -p 172.17.68.16:3260 --rescan from (pid=1013) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:11:41,215 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 2, target: iqn.2010-10.org.openstack:volume-00000008, portal: 172.17.68.16,3260] stderr= from (pid=1013) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:11:45,218 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-172.17.68.16:3260-iscsi-iqn.2010-10.org.openstack:volume-00000008-lun-0. Will rescan & retry. Try number: 2 2011-10-18 14:11:45,218 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000008 -p 172.17.68.16:3260 --rescan from (pid=1013) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:11:45,273 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 2, target: iqn.2010-10.org.openstack:volume-00000008, portal: 172.17.68.16,3260] stderr= from (pid=1013) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:11:54,278 ERROR nova.rpc [-] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume (nova.rpc): TRACE: volume_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume (nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 551, in discover_volume (nova.rpc): TRACE: (mount_device)) (nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-172.17.68.16:3260-iscsi-iqn.2010-10.org.openstack:volume-00000008-lun-0 (nova.rpc): TRACE: root@node1:~# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2010-10.org.openstack:volume-00000008 System information: Driver: iscsi State: ready I_T nexus information: I_T nexus: 1 Initiator: iqn.1993-08.org.debian:01:dc2ba8e71e8e Connection: 0 IP Address: 172.17.68.100 <========== OK (node2 initiator) LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 1074 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/nova-volumes/volume-00000008 Backing store flags: Account information: ACL information: ALL root@node2:/dev/disk/by-path# ls -l lrwxrwxrwx 1 root root 9 2011-10-18 14:11 ip-172.17.68.16:3260-iscsi-iqn.2010-10.org.openstack:volume-00000008-lun-1 -> ../../sda <========== OK (lun-1) Problem: still looking for lun-0 when it should look for lun-1 (nova-compute problem) ******************************************************************************************** B) instance and volume on node2: OK ******************************************************************************************** root@client:~# euca-create-volume -s 1 -z nova VOLUME vol-00000009 1 creating (project-one, None, None, None) 2011-10-18T12:03:31Z root@client:~# euca-describe-volumes VOLUME vol-00000009 1 nova available (project-one, node2, None, None) 2011-10-18T12:03:31Z root@node2:~# lvdisplay --- Logical volume --- LV Name /dev/nova-volumes/volume-00000009 VG Name nova-volumes LV UUID N89fbU-xr2N-jhfc-UHJD-fLwY-m1US-vtSiCm LV Write Access read/write LV Status available # open 1 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:2 root@node2:~# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2010-10.org.openstack:volume-00000009 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 1074 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/nova-volumes/volume-00000009 Backing store flags: Account information: ACL information: ALL root@client:~# euca-run-instances -k key-dodeeric -t m1.medium ami-00000002 root@client:~# euca-describe-instances RESERVATION r-sr4ftljp project-one default INSTANCE i-00000004 ami-00000002 10.0.1.3 10.0.1.3 running key-dodeeric (project-one, node2) 0 m1.medium 2011-10-18T12:09:42Z nova aki-00000001 ami-00000000 root@client:~# euca-attach-volume -i i-00000004 -d /dev/vdd vol-00000009 VOLUME vol-00000009 root@client:~# euca-describe-volumes VOLUME vol-00000009 1 nova in-use (project-one, node2, i-00000004[node2], /dev/vdd) 2011-10-18T12:03:31Z <========== OK root@node1:~# ssh -i key-dodeeric.priv ubuntu@10.0.1.3 Welcome to Ubuntu 11.10 (GNU/Linux 3.0.0-12-virtual x86_64) ubuntu@server-4:~$ sudo fdisk -l Disk /dev/vdc: 1073 MB, 1073741824 bytes <========== OK (disk is seen from inside the instance) root@node2:~# tail -200 /var/log/nova/nova-compute.log 2011-10-18 14:20:11,571 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'5bbce3c8-4910-4bad-89f4-763d58592a50', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdd', u'volume_id': 9}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-18T12:20:11.473531', u'_context_user_id': u'dodeeric', u'method': u'attach_volume', u'_context_remote_address': u'172.17.68.16'} from (pid=1013) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2011-10-18 14:20:11,571 DEBUG nova.rpc [-] unpacked context: {'user_id': u'dodeeric', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-18T12:20:11.473531', 'auth_token': None, 'msg_id': None, 'remote_address': u'172.17.68.16', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'5bbce3c8-4910-4bad-89f4-763d58592a50', 'project_id': u'project-one', 'read_deleted': False} from (pid=1013) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2011-10-18 14:20:11,572 INFO nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] check_instance_lock: decorating: || 2011-10-18 14:20:11,573 INFO nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] check_instance_lock: arguments: || || |4| 2011-10-18 14:20:11,573 DEBUG nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] instance 4: getting locked state from (pid=1013) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165 2011-10-18 14:20:11,638 INFO nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] check_instance_lock: locked: |False| 2011-10-18 14:20:11,638 INFO nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] check_instance_lock: admin: |True| 2011-10-18 14:20:11,638 INFO nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] check_instance_lock: executing: || 2011-10-18 14:20:11,705 AUDIT nova.compute.manager [5bbce3c8-4910-4bad-89f4-763d58592a50 dodeeric project-one] instance 4: attaching volume 9 to /dev/vdd root@node2:~# tgtadm --lld iscsi --op show --mode target <========== OK (no initiator connected: iSCSI not used if instance and volume on the same node) Target 1: iqn.2010-10.org.openstack:volume-00000009 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 1074 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/nova-volumes/volume-00000009 Backing store flags: Account information: ACL information: ALL ******************************************************************************************** C) instance on node1 and volume on node2: NOK ******************************************************************************************** root@client:~# euca-create-volume -s 1 -z nova VOLUME vol-0000000a 1 creating (project-one, None, None, None) 2011-10-18T12:30:46Z root@client:~# euca-describe-volumes VOLUME vol-0000000a 1 nova available (project-one, node2, None, None) 2011-10-18T12:30:46Z root@node2:~# lvdisplay --- Logical volume --- LV Name /dev/nova-volumes/volume-0000000a VG Name nova-volumes LV UUID HCdftL-bTeB-KwNE-yb83-mqKT-eO9d-IuHC2v LV Write Access read/write LV Status available # open 1 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:3 root@node2:~# tgtadm --lld iscsi --op show --mode target Target 2: iqn.2010-10.org.openstack:volume-0000000a System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00020000 SCSI SN: beaf20 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00020001 SCSI SN: beaf21 Size: 1074 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/nova-volumes/volume-0000000a Backing store flags: Account information: ACL information: ALL root@client:~# euca-run-instances -k key-dodeeric -t m1.medium ami-00000002 RESERVATION r-tmxplzvs project-one default INSTANCE i-00000008 ami-00000002 pending key-dodeeric (project-one, None) 0 m1.medium 2011-10-18T12:37:25Z unknown zone aki-00000001 ami-00000000 root@client:~# euca-describe-instances INSTANCE i-00000008 ami-00000002 10.0.1.7 10.0.1.7 running key-dodeeric (project-one, node1) 0 m1.medium 2011-10-18T12:37:25Z nova aki-00000001 ami-00000000 root@client:~# euca-attach-volume -i i-00000008 -d /dev/vdd vol-0000000a VOLUME vol-0000000a root@client:~# euca-describe-volumes VOLUME vol-0000000a 1 nova available (project-one, node2, None, None) 2011-10-18T12:30:46Z <========== NOK (not attached) root@node2:~# tgtadm --lld iscsi --op show --mode target Target 2: iqn.2010-10.org.openstack:volume-0000000a System information: Driver: iscsi State: ready I_T nexus information: I_T nexus: 1 Initiator: iqn.1993-08.org.debian:01:2e5ee63e3872 Connection: 0 IP Address: 172.17.68.16 <========== OK (node1 initiator) LUN information: LUN: 0 Type: controller SCSI ID: IET 00020000 SCSI SN: beaf20 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00020001 SCSI SN: beaf21 Size: 1074 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/nova-volumes/volume-0000000a Backing store flags: Account information: ACL information: ALL root@node1:/dev/disk/by-path# ls -l lrwxrwxrwx 1 root root 9 2011-10-18 14:39 ip-172.17.68.100:3260-iscsi-iqn.2010-10.org.openstack:volume-0000000a-lun-1 -> ../../sda <========== OK root@node1:~# tail -200 /var/log/nova/nova-compute.log 2011-10-18 14:39:25,519 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'0e1d4934-97f8-4115-83aa-7d08f5e92f32', u'_context_read_deleted': False, u'args': {u'instance_id': 8, u'mountpoint': u'/dev/vdd', u'volume_id': 10}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-18T12:39:25.401601', u'_context_user_id': u'dodeeric', u'method': u'attach_volume', u'_context_remote_address': u'172.17.68.16'} from (pid=1743) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2011-10-18 14:39:25,519 DEBUG nova.rpc [-] unpacked context: {'user_id': u'dodeeric', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-18T12:39:25.401601', 'auth_token': None, 'msg_id': None, 'remote_address': u'172.17.68.16', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'0e1d4934-97f8-4115-83aa-7d08f5e92f32', 'project_id': u'project-one', 'read_deleted': False} from (pid=1743) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2011-10-18 14:39:25,520 INFO nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] check_instance_lock: decorating: || 2011-10-18 14:39:25,521 INFO nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] check_instance_lock: arguments: || || |8| 2011-10-18 14:39:25,521 DEBUG nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] instance 8: getting locked state from (pid=1743) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165 2011-10-18 14:39:25,648 INFO nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] check_instance_lock: locked: |False| 2011-10-18 14:39:25,649 INFO nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] check_instance_lock: admin: |True| 2011-10-18 14:39:25,649 INFO nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] check_instance_lock: executing: || 2011-10-18 14:39:25,712 AUDIT nova.compute.manager [0e1d4934-97f8-4115-83aa-7d08f5e92f32 dodeeric project-one] instance 8: attaching volume 10 to /dev/vdd 2011-10-18 14:39:25,753 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery 2011-10-18 14:39:25,753 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node2 from (pid=1743) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:39:25,967 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 172.17.68.100:3260,1 iqn.2010-10.org.openstack:volume-0000000a from (pid=1743) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479 2011-10-18 14:39:25,970 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000000a -p 172.17.68.100:3260 --login from (pid=1743) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:39:26,576 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-0000000a, portal: 172.17.68.100,3260] Login to [iface: default, target: iqn.2010-10.org.openstack:volume-0000000a, portal: 172.17.68.100,3260]: successful stderr= from (pid=1743) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:39:26,581 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000000a -p 172.17.68.100:3260 --op update -n node.startup -v automatic from (pid=1743) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:39:26,667 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1743) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:39:26,669 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-172.17.68.100:3260-iscsi-iqn.2010-10.org.openstack:volume-0000000a-lun-0. Will rescan & retry. Try number: 0 2011-10-18 14:39:26,679 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000000a -p 172.17.68.100:3260 --rescan from (pid=1743) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:39:26,783 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 2, target: iqn.2010-10.org.openstack:volume-0000000a, portal: 172.17.68.100,3260] stderr= from (pid=1743) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:39:27,786 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-172.17.68.100:3260-iscsi-iqn.2010-10.org.openstack:volume-0000000a-lun-0. Will rescan & retry. Try number: 1 2011-10-18 14:39:27,787 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000000a -p 172.17.68.100:3260 --rescan from (pid=1743) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:39:27,845 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 2, target: iqn.2010-10.org.openstack:volume-0000000a, portal: 172.17.68.100,3260] stderr= from (pid=1743) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:39:31,849 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-172.17.68.100:3260-iscsi-iqn.2010-10.org.openstack:volume-0000000a-lun-0. Will rescan & retry. Try number: 2 2011-10-18 14:39:31,850 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000000a -p 172.17.68.100:3260 --rescan from (pid=1743) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168 2011-10-18 14:39:31,902 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 2, target: iqn.2010-10.org.openstack:volume-0000000a, portal: 172.17.68.100,3260] stderr= from (pid=1743) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506 2011-10-18 14:39:40,908 ERROR nova.rpc [-] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume (nova.rpc): TRACE: volume_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume (nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 551, in discover_volume (nova.rpc): TRACE: (mount_device)) (nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-172.17.68.100:3260-iscsi-iqn.2010-10.org.openstack:volume-0000000a-lun-0 (nova.rpc): TRACE: --- On a platform where both nodes are running natty and iet (and not tgt), nova-compute searchs for lun-0 and find it: the volume can be attached correctly to the instance: root@node1:~# euca-describe-instances RESERVATION r-jzwo7bdc project-toc group-toc INSTANCE i-0000001c ami-00000019 192.168.1.240 10.0.3.3 running key-dodeeric (project-toc, node1) 0 a1.medium 2011-10-14T08:35:23Z nova aki-00000005 ari-00000006 root@node1:~# euca-describe-volumes VOLUME vol-00000004 5 nova in-use (project-toc, node2, i-0000001c[node1], /dev/vdd) 2011-10-11T10:13:07Z root@node1:/var/log/nova# zgrep -i "2011-10-14 10:40:5" nova-compute.log.4 2011-10-14 10:40:58,648 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'0ce80ca4-c77f-40e7-8366-9bdcc14f1c32', u'_context_read_deleted': False, u'args': {u'instance_id': 28, u'mountpoint': u'/dev/vdd', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': False, u'_context_project_id': u'project-toc', u'_context_timestamp': u'2011-10-14T08:40:58.552407', u'_context_user_id': u'dodeeric', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=2608) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2011-10-14 10:40:58,648 DEBUG nova.rpc [-] unpacked context: {'user_id': u'dodeeric', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-14T08:40:58.552407', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': False, 'request_id': u'0ce80ca4-c77f-40e7-8366-9bdcc14f1c32', 'project_id': u'project-toc', 'read_deleted': False} from (pid=2608) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2011-10-14 10:40:58,649 INFO nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] check_instance_lock: decorating: || 2011-10-14 10:40:58,650 INFO nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] check_instance_lock: arguments: || || |28| 2011-10-14 10:40:58,650 DEBUG nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] instance 28: getting locked state from (pid=2608) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165 2011-10-14 10:40:58,709 INFO nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] check_instance_lock: locked: |False| 2011-10-14 10:40:58,709 INFO nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] check_instance_lock: admin: |False| 2011-10-14 10:40:58,709 INFO nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] check_instance_lock: executing: || 2011-10-14 10:40:58,745 AUDIT nova.compute.manager [0ce80ca4-c77f-40e7-8366-9bdcc14f1c32 dodeeric project-toc] instance 28: attaching volume 4 to /dev/vdd 2011-10-14 10:40:58,762 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery 2011-10-14 10:40:58,762 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node2 from (pid=2608) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165 2011-10-14 10:40:58,793 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.202:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=2608) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:487 2011-10-14 10:40:58,794 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.202:3260 --login from (pid=2608) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165 2011-10-14 10:40:59,323 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.202,3260] 2011-10-14 10:40:59,324 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.202:3260 --op update -n node.startup -v automatic from (pid=2608) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165 2011-10-14 10:40:59,345 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=2608) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:514 2011-10-14 10:40:59,346 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.202:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0 2011-10-14 10:40:59,346 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.202:3260 --rescan from (pid=2608) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165 2011-10-14 10:40:59,366 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 1, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.202,3260] 2011-10-14 10:40:59,367 DEBUG nova.volume.driver [-] Found iSCSI node /dev/disk/by-path/ip-192.168.1.202:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0 (after 1 rescans) from (pid=2608) discover_volume /usr/lib/python2.7/dist-packages/nova/volume/driver.py:570