I cannot attach a volume to an instance if both are on different hosts.

This a problem which is specific to tgt which is used on Oneiric. I file that bug in Nova (Ubuntu) and not in Nova (Openstack), even though it is not an Ubuntu packaging problem. 

Example:

SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)

node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004

root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc vol-00000004

Error seen in nova-compute.log:

Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0

Explanation: nova-compute is waiting for a device to be configured as 

/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1

but the device is configued as
 /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0

nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:

root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root  9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd

I guess this is a difference between the iet and tgt softwares used to configure iSCSI targets.

On Oneiric, tgt is used by default, and iet cannot be used because the kernel module is no more available.

In nova.conf:

--iscsi_helper=tgtadm

If I adapt the code from lun-0 to lun-1, everything works well:

diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
<         mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
>         mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %

Complete nova-compute.log:

root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': 

{u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', 

u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ 

/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 

'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) 

_unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |

<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-

packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-

packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute 

/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
 stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from 

(pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-

packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry.  Try 

number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute 

/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
 stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry.  Try 

number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute 

/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
 stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry.  Try 

number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute 

/usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
 stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE:     rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE:     function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE:     volume_id)
(nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE:     path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE:     (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status

root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root  9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root  9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root  9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0

root@node1:~#  tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1993-08.org.debian:01:63503b146b7  
            Connection: 0
                IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1 <==================================== LUN = 1 (and not 0)
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/nova-volumes/volume-00000004
            Backing store flags:
    Account information:
    ACL information:
        ALL