Unable to attache more than one volume to a compute-node

[Razique Mahroua]

Hi,
I've a server running nova-volumes and a compute-node. Both servers are running Ubuntu 10.04 LTS.
The compute-node has open-iscsi package installed on it.

When I run an euca-attach volume for an instance, it works fine.
Now, If I try to attache another volume to another instance, i've the following trace into nova-compute.log :
http://paste.openstack.org/

The first time I try to attach it, there is no error, but the line :

2011-06-22 11:49:28,807 AUDIT nova.compute.manager [AX678B9WGP8DERIYVTP5 nuage-and-co nuage-and-co] instance 136: attaching volume 10 to \/dev\/sdg

When I check the volume attachment via euca-describe volumes :
VOLUME vol-0000000a 50 nova available (nuage, nova-cc1, None, None) 2011-06-22T09:47:42Z

So it remains available. So I try a second time, but then I've the iscsi error that you can see into my paste.
If I look at the portal :

iSCSI Transport Class version 2.0-870
version 2.0-871
Target: iqn.2010-10.org.openstack:volume-00000007
 Current Portal: 172.16.40.244:3260,1
 Persistent Portal: 172.16.40.244:3260,1
  **********
  Interface:
  **********
  Iface Name: default
  Iface Transport: tcp
  Iface Initiatorname: iqn.1993-08.org.debian:01:a82929197c4b
  Iface IPaddress: 172.16.40.247
  Iface HWaddress: (null)
  Iface Netdev: (null)
  SID: 2
  iSCSI Connection State: LOGGED IN
  iSCSI Session State: LOGGED_IN
  Internal iscsid Session State: NO CHANGE
  ************************
  Negotiated iSCSI params:
  ************************
  HeaderDigest: None
  DataDigest: None
  MaxRecvDataSegmentLength: 262144
  MaxXmitDataSegmentLength: 8192
  FirstBurstLength: 65536
  MaxBurstLength: 262144
  ImmediateData: Yes
  InitialR2T: Yes
  MaxOutstandingR2T: 1
  ************************
  Attached SCSI devices:
  ************************
  Host Number: 1 State: running
  scsi1 Channel 00 Id 0 Lun: 0
   Attached scsi disk sda State: running

And the discovery mode returns the following paths :

root@nova-cn1:~# iscsiadm -m discovery -t st -p 172.16.40.244
192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-00000004
172.16.40.77:3260,1 iqn.2010-10.org.openstack:volume-00000004
172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-00000004
172.16.40.76:3260,1 iqn.2010-10.org.openstack:volume-00000004
10.0.1.1:3260,1 iqn.2010-10.org.openstack:volume-00000004
172.16.40.245:3260,1 iqn.2010-10.org.openstack:volume-00000004
169.254.169.254:3260,1 iqn.2010-10.org.openstack:volume-00000004
172.16.40.78:3260,1 iqn.2010-10.org.openstack:volume-00000004
192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-00000007
172.16.40.77:3260,1 iqn.2010-10.org.openstack:volume-00000007
172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-00000007
172.16.40.76:3260,1 iqn.2010-10.org.openstack:volume-00000007
10.0.1.1:3260,1 iqn.2010-10.org.openstack:volume-00000007
172.16.40.245:3260,1 iqn.2010-10.org.openstack:volume-00000007
169.254.169.254:3260,1 iqn.2010-10.org.openstack:volume-00000007
172.16.40.78:3260,1 iqn.2010-10.org.openstack:volume-00000007
192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-00000009
172.16.40.77:3260,1 iqn.2010-10.org.openstack:volume-00000009
172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-00000009
172.16.40.76:3260,1 iqn.2010-10.org.openstack:volume-00000009
10.0.1.1:3260,1 iqn.2010-10.org.openstack:volume-00000009
172.16.40.245:3260,1 iqn.2010-10.org.openstack:volume-00000009
169.254.169.254:3260,1 iqn.2010-10.org.openstack:volume-00000009
172.16.40.78:3260,1 iqn.2010-10.org.openstack:volume-00000009
192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.77:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.76:3260,1 iqn.2010-10.org.openstack:volume-0000000a
10.0.1.1:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.245:3260,1 iqn.2010-10.org.openstack:volume-0000000a
169.254.169.254:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.78:3260,1 iqn.2010-10.org.openstack:volume-0000000a

Thanks

[Razique Mahroua (razique)]

Sorry,
here is the good paste :
http://paste.openstack.org/show/1733/

[Razique Mahroua (razique)]

Ok, after I ran an analyse, I figured that due to the iscsi multipathing, during the discovery, the target uses instances's public ip as an iscsi path :

192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.77:3260,1 iqn.2010-10.org.openstack:volume-0000000a <---- FLOATING IP
172.16.40.244:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.76:3260,1 iqn.2010-10.org.openstack:volume-0000000a
10.0.1.1:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.245:3260,1 iqn.2010-10.org.openstack:volume-0000000a
169.254.169.254:3260,1 iqn.2010-10.org.openstack:volume-0000000a
172.16.40.78:3260,1 iqn.2010-10.org.openstack:volume-0000000a <----- FLOATING IP

theses ip are used during the iscsi login :
'iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-0000000a, portal: 172.16.40.77,3260

But it won't work since this IP is incorrect ; is it possible to exclude floating ips as paths from the iscsi target ?

[Razique Mahroua (razique)]

I figured it out!
You need to dedicate a network for the iscsi traffic, if you want the multipathing; or set explicitely the exact ip adress you would like to use :
--iscsi_ip_prefix=$cloud_ip or $whatever
do no forget to set it on the compute-node(s)' conf and on the server which runs the nova-volume service