cannot attach a volume to an instance

Bug #615646 reported by C de-Avillez on 2010-08-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
Invalid
Undecided
Unassigned
eucalyptus (Ubuntu)
High
Dave Walker
Maverick
High
Dave Walker

Bug Description

ubuntu@cempedak:~$ euca-run-instances -g test-g0 -k test -t m1.xlarge -z UEC-TEST1 emi-3D891C6C
RESERVATION r-41FB0787 admin admin-test-g0
INSTANCE i-3AD9069B emi-3D891C6C 0.0.0.0 0.0.0.0 pending test 0 m1.xlarge 2010-08-10T00:40:42.55Z UEC-TEST1 eki-DA752171
ubuntu@cempedak:~$ euca-describe-instances
RESERVATION r-41FB0787 admin test-g0
INSTANCE i-3AD9069B emi-3D891C6C 10.55.55.100 172.19.1.2 running test 0 m1.xlarge 2010-08-10T00:40:42.55Z UEC-TEST1 eki-DA752171
ubuntu@cempedak:~$ euca-describe-volumes
VOLUME vol-5A280632 1 UEC-TEST1 available 2010-08-10T00:39:22.623Z
ubuntu@cempedak:~$ euca-attach-volume -i i-3AD9069B -d /dev/sdb vol-5A280632
VOLUME vol-5A280632
ubuntu@cempedak:~$ euca-describe-volumes
VOLUME vol-5A280632 1 UEC-TEST1 available 2010-08-10T00:39:22.623Z
ubuntu@cempedak:~$ euca-describe-instances
RESERVATION r-41FB0787 admin test-g0
INSTANCE i-3AD9069B emi-3D891C6C 10.55.55.100 172.19.1.2 running test 0 m1.xlarge 2010-08-10T00:40:42.55Z UEC-TEST1 eki-DA752171
ubuntu@cempedak:~$ euca-describe-volumes
VOLUME vol-5A280632 1 UEC-TEST1 available 2010-08-10T00:39:22.623Z
ubuntu@cempedak:~$ euca-describe-instances
RESERVATION r-41FB0787 admin test-g0
INSTANCE i-3AD9069B emi-3D891C6C 10.55.55.100 172.19.1.2 running test 0 m1.xlarge 2010-08-10T00:40:42.55Z UEC-TEST1 eki-DA752171
ubuntu@cempedak:~$

euca2ools 1.2-0ubuntu10 install ok installed
eucalyptus-cc 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
eucalyptus-cloud 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
eucalyptus-common 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
eucalyptus-gl 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
eucalyptus-java-common 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
eucalyptus-sc 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
eucalyptus-walrus 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed
uec-component-listener 2.0~bzr1219-0ubuntu1 eucalyptus install ok installed

C de-Avillez (hggdh2) wrote :
Download full text (5.2 KiB)

This is an all-in-one install. cloud-debug excerpt:

20:39:22 DEBUG [Context:New I/O server worker #1-5] :1281400762553:Context:CONTEXT_CREATE:5b2eaee5-20f1-40e8-a75b-4e22159a5962:[id: 0x7d26c5d4, /10.55.55.2:51634 => /10.55.55.2:8773]:NioHttpDecoder.decode.129
20:39:22 DEBUG [TimedEvictionSet:New I/O server worker #1-5] Evicting previous entry: FMd6P2GxVBMlg7PpCHqRt1/5x6OOUaW3TQubZSzsRdQ 5940707576109
20:39:22 DEBUG [Context:New I/O server worker #1-5] :1281400762573:Context:CONTEXT_USER:5b2eaee5-20f1-40e8-a75b-4e22159a5962:admin:NativeMethodAccessorImpl.invoke0.-2
20:39:22 DEBUG [Context:New I/O server worker #1-5] :1281400762574:Context:CONTEXT_SUBJECT:5b2eaee5-20f1-40e8-a75b-4e22159a5962:[DatabaseWrappedUser [ user = UserEntity [ administrator = true, enabled = true, name = admin, password = 21232f297a57a5a743894a0e4a801fc3, queryId = WKy3rMzOWPouVOxK1p3Ar1C2uRBwa2FBXnCw, s
ecretKey = DteXlXLRBPjDIVWaAQQ7nWaGiPjbamP1K6FIsA, token = YR7SQsocoiC5JH4C7ltUOPMOdtgY36GnkbSsiXrejIFCoS9N6at9hXVMkFVdQ7bk2nWJBdTC009jFdX1Tg, certificates = 33908414070706 null LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGVENDQWYyZ0F3SUJBZ0lHSHRicStiK3lNQTBHQ1NxR1NJYjNEUUVCRFFVQU1FRXhDekFKQmdOVkJBWVQKQWxWVE1RMHdDd1l
EVlFRS0V3UlZjMlZ5TVJNd0VRWURWUVFMRXdwRmRXTmhiSGx3ZEhWek1RNHdEQVlEVlFRRApFd1ZoWkcxcGJqQWVGdzB4TURBNE1Ea3lNekk0TkRWYUZ3MHhOVEE0TURreU16STRORFZhTUVFeEN6QUpCZ05WCkJBWVRBbFZUTVEwd0N3WURWUVFLRXdSVmMyVnlNUk13RVFZRFZRUUxFd3BGZFdOaGJIbHdkSFZ6TVE0d0RBWUQKVlFRREV3VmhaRzFwYmpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dn
RUJBS0tvWG9pYgp6cU5FdjJ6QVNQd1BDN1JtMGpDcEl0SEd1dStObm9tQlpuKzZ0WmhGVy9CdEJFRWZEYy92aERmcm9mZ0FONDBJCkRJejE3UW8wVXVyeEFlaXN3czVadVoySHNFQnlGdUZFSHlPSnk4aXVqMVBYbmRza3BSOHRxTzZCK0JhT1RhVnAKN0dVVmtRL09hMlZvWk5TWVR2Ti9NbU5Ud2JRSklIVUl1Y2VqUXRwbGFOZDdiQzB4QW03OEVvTkhqNHVRQVdMeAp2dmFFVWZsRXV0UTAzTGlaUityekFPNTFqVjgxakhLR
HVOb0FFdTR1QTBiajFGZ0k4V25ibUhORXJrVVFGazZHCnQ0Z1FwbVlRaE9hQ3d0aDYyWVJ6UjFia0hUT0JpUjNZL25QN0hDRFJ1bm9HOFBPSXVieEhMbjJZbXlDNTdkNFMKNWYxcGFvckRnekZWd1YwQ0F3RUFBYU1UTUJFd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBTkJna3Foa2lHOXcwQgpBUTBGQUFPQ0FRRUFNTVRoekNiS0pWZmgycTRuNDQ4SHNaa2ROZlNnRGJKZzJEY0FONXVpL2dPdjRCMG1BODdxCjJxVFlaSXQza0
9zcTM5OURBUzdHRS9KcWgwODMvSjgzTWt6TjY5QURrS3NzV2Z6SmExR0JvTzRYU2Y3R1g5RUoKaEI5ZlhpUnV3ZlVBV3RCTW01aVd6UjlWbFh6a0pqT0VtTUZYUWlsTXB3cmtCeWM2Vm5DMk96LzhnczBqbjc2OApsY0xGblFlWTR6TU90WXlEZk1GMFdrWGtlL3E3dHBoNjZHY3hFMFBINXNodElpdDVRMDJkTjJYWTdxTHlFU1I1CjFPdVhFWHdTWlFjemtPdE9mekl5MFVaVmdGRHRhSHQxeCtxTndZeVBCQmxzZVROK2JVbGl
vRjgyblFTdFZxOVMKT2M3dEpBZUEvbERycm9WVm8yU2p3b0IrQ3JhODJKQjBJQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K, eucaGroupIds = ], ], com.eucalyptus.auth.DatabaseWrappedGroup@423dca93]:NativeMethodAccessorImpl.invoke0.-2
20:39:22 DEBUG [RequestQueue:RequestQueue.2] :1281400762600:RequestQueue:5b2eaee5-20f1-40e8-a75b-4e22159a5962:admin:MSG_RECEIVED:5b2eaee5-20f1-40e8-a75b-4e22159a5962:CreateVolumeType:NativeMethodAccessorImpl.invoke0.-2
20:39:22 INFO [VolumeManager:Volume.2] :1281400762711:VolumeManager:5b2eaee5-20f1-40e8-a75b-4e22159a5962:admin:VOLUME_CREATE:user=admin:volume=vol-5A280632:size=1:cluster=UEC-TEST1:snapshot=null:NativeMethodAccessorImpl.invoke0.-2
20:39:22 DEBUG [SystemUtil:pool-8-thread-1] Running comma...

Read more...

C de-Avillez (hggdh2) wrote :

ubuntu@cempedak:/var/log/eucalyptus$ sudo losetup -a
/dev/loop0: [6801]:3146674 (/var/lib/eucalyptus/volumes/vol-5A280632)
ubuntu@cempedak:/var/log/eucalyptus$ sudo vg
vgcfgbackup vgcfgrestore vgchange vgck vgconvert vgcreate vgdisplay vgexport vgextend vgimport vgimportclone vgmerge vgmknodes vgreduce vgremove vgrename vgs vgscan vgsplit
ubuntu@cempedak:/var/log/eucalyptus$ sudo vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg-E1rO0g.." using metadata type lvm2
ubuntu@cempedak:/var/log/eucalyptus$ sudo lvscan
  ACTIVE '/dev/vg-E1rO0g../lv-rNkFoQ..' [1.00 GiB] inherit
ubuntu@cempedak:/var/log/eucalyptus$

Mathias Gug (mathiaz) on 2010-08-10
Changed in eucalyptus (Ubuntu):
importance: Undecided → Medium
C de-Avillez (hggdh2) wrote :

Setting Importance to critical -- this has to be looked at before feature freeze, and it is a major issue on using the cloud.

Changed in eucalyptus (Ubuntu):
importance: Medium → Critical
C de-Avillez (hggdh2) wrote :

Actually HIGH. No need to poke the release team...

Changed in eucalyptus (Ubuntu):
importance: Critical → High
Thierry Carrez (ttx) on 2010-08-11
Changed in eucalyptus (Ubuntu Maverick):
milestone: none → ubuntu-10.10-beta
assignee: nobody → Dave Walker (davewalker)
C de-Avillez (hggdh2) wrote :

With r1224, still a problem. We *did* advance a bit:

18:39:41 INFO [VolumeManager:Volume.16] :1281566381374:VolumeManager:89a38b73-8783-4376-a403-fa0c569f193f:admin:VOLUME_CREATE:user=admin:volume=vol-5A010635:size=1:cluster=UEC-TEST1:snapshot=null:NativeMethodAccessorImpl.invoke0.-2
18:39:41 DEBUG [SystemUtil:pool-8-thread-1] Running command: ///usr/lib/eucalyptus/euca_rootwrap dd if=/dev/zero of=//var/lib/eucalyptus/volumes/vol-5A010635 count=1 bs=1M seek=1027
18:39:41 INFO [OverlayManager:pool-8-thread-1] losetup /dev/loop0 //var/lib/eucalyptus/volumes/vol-5A010635
18:40:11 DEBUG [QueuedEventCallback:Volume.16] :1281566411345:QueuedEventCallback:bc6d011b-594b-4150-a9c3-485998fc03d4:admin:QUEUE:class edu.ucsb.eucalyptus.msgs.AttachVolumeType:[AttachVolumeType volumeId=vol-5A010635 instanceId=i-48AF0842 device=/dev/sdb remoteDevice=//,10.55.55.2,iqn.2009-06.com.eucalyptus.UEC-TEST1:store1,AG0f08WwmtrqkqS8+fHiasNQKaWqp3Rbgv9NqSfdWfbEu6RX5UFk9OOE5gGKNlj68/1h3dXXXRYIv3kTtNRTBw+nl0l3M0owNm9CWU0+IMmjbitjdRfvjQkTZg5Zz/eK7PjSnoGbRHKrwvnC/FH82rAuf6enq8tTeM+EdVXkeLKC2KGcrCIGR7V6ec1Snzcu33KClX5+A7DG24V7DEe1Jwos/zES1ciOJaS9K6XbiozjtXIhj7zu9jNcj6twS3uZuHzveCZcK9ZmWJygnELDNvg0pRy5eZ/o+6zzsj/Pk6G0wM6tLnLNs828I0h0uOHpZjOUhUoTGkjG4r2C/CY8wg== correlationId=bc6d011b-594b-4150-a9c3-485998fc03d4 userId=admin effectiveUserId=eucalyptus _return=false statusMessage=null]:NativeMethodAccessorImpl.invoke0.-2
18:40:11 INFO [VolumeManager:Volume.16] :1281566411346:VolumeManager:bc6d011b-594b-4150-a9c3-485998fc03d4:admin:VOLUME_ATTACH:user=admin:volume=vol-5A010635:instance=i-48AF0842:cluster=UEC-TEST1:NativeMethodAccessorImpl.invoke0.-2
  <volumeId>vol-5A010635</volumeId>
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Trying to remove invalid volume attachment vol-5A010635 from instance i-48AF0842
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Removed failed attachment: vol-5A010635 -> i-48AF0842
  <volumeId>vol-5A010635</volumeId>
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Trying to remove invalid volume attachment vol-5A010635 from instance i-48AF0842
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Removed failed attachment: vol-5A010635 -> i-48AF0842

Dave Walker (davewalker) on 2010-08-12
Changed in eucalyptus (Ubuntu Maverick):
status: New → Confirmed
Ye Wen (wenye) wrote :

The problem is that the iscsi related commands and scripts are not listed in the /etc/eucalyptus/wrappers.conf on NC. I guess there is some misunderstanding between both our sides. We assume you know that all the original sudoers config needs to be moved into wrappers.conf. But obviously this information was not conveyed clearly.

So, here is the stuff you need to add to the wrappers.conf on NC:

get_iscsitarget.pl /usr/share/eucalyptus/get_iscsitarget.pl 0
connect_iscsitarget.pl /usr/share/eucalyptus/connect_iscsitarget.pl 0
disconnect_iscsitarget.pl /usr/share/eucalyptus/disconnect_iscsitarget.pl 0

And put the following in wrappers.conf on SC:

tgtadm /usr/sbin/tgtadm 0

Or maybe just list the 4 items in all wrappers.conf for easiness.

Also, remember to attach a device using "euca-attach-volume ... -d /dev/vdx" since SCSI hotplug is not working on Maverick anymore.

I have successfully tested the process in our Maverick daily build installation. Please give it try again by following the procedure I listed above and let us know if you still have problems with EBS attaching.

Given there is no work from Eucalyptus side, and all you have to do is a change in the config, I'm closing this bug as fixed now. Please reopen it if there is still problem. But please open another bug if the problem is different.

Changed in eucalyptus:
status: New → Invalid
Dave Walker (davewalker) wrote :

@Ye, Thanks for that update.. i'll get that integrated now!

Dave Walker (davewalker) on 2010-08-12
Changed in eucalyptus (Ubuntu Maverick):
status: Confirmed → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 2.0~bzr1230-0ubuntu1

---------------
eucalyptus (2.0~bzr1230-0ubuntu1) maverick; urgency=low

  * New upstream merge, r1230 - bug fixes:
    - (LP: #617127, #611769, #617032, #613033, #613033, #584497)
  * debian/patches/18-priv_security.patch: Added *_iscsitarget.pl scripts to
    the wrappers.conf whitelist. (LP: #615646)
  * debian/patches/15-fix-default-ramdisk.patch: Dropped, fixed applied upstream.
 -- Dave Walker (Daviey) <email address hidden> Fri, 13 Aug 2010 11:25:25 +0100

Changed in eucalyptus (Ubuntu Maverick):
status: Fix Committed → Fix Released
Fernando Abramowitz (ferabra) wrote :

Hi, I installed a maverick UEC. I have 1 system with clc,cc,sc,walrus and another with nc. when attach a volume to a running instance no error is show but a can't see the second disk with fdisk -l. I verify wrappers.conf file and everything is ok. I need to open a new bug?

las line of cc.log

print_ccInstance(): refresh_instances(): instanceId=i-402607EE reservationId=r-3F800688 emiId=emi-DF71106F kernelId=eki-F5B510F4 ramdiskId=eri-0A2C115D emiURL=http://192.168.3.252:8773/services/Walrus/image-store-1295465260/image.manifest.xml kernelURL=http://192.168.3.252:8773/services/Walrus/image-store-1295465260/kernel.manifest.xml ramdiskURL=http://192.168.3.252:8773/services/Walrus/image-store-1295465260/ramdisk.manifest.xml state=Extant ts=1295546050 ownerId=fabramowitz keyName=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyso3nDkuOJX2S+g6Av2npyM10LNemRWR2CLRmLYNguLHxsAQznATaQqgGXaxV/now4vY22t++ARMSEW3eQM5wgHbawIbRm40JMzU021nzsE1lI9SF9whQkzGn7FAFI6wezRV35f2PA/10t05WanE8zn31unUo1aO5WExiWoeNDSNfD19wBpHLk0tG3ZLs3kxPhYK+qfVSLJdNmztYIRPoJsDdFlIogoXycDb2iy/pq0HbO6E8MRFa61VNRTs+5HdZCeQsTY/AZG3nmk5vlOIAntZ2IV3pvfwNVzBZncEE4rb4Qd5pIwJvBrpnF1Y91NDmStAuH9ddrepH1lOgdsAB fabramowitz@eucalyptus ccnet={privateIp=172.19.1.2 publicIp=192.168.3.50 privateMac=D0:0D:40:26:07:EE vlan=10 networkIndex=2} ccvm={cores=2 mem=512 disk=10} ncHostIdx=0 serviceTag=http://10.98.98.2:8775/axis2/services/EucalyptusNC userData= launchIndex=0 volumesSize=1 volumes={vol-A37808CF,//,192.168.3.252,iqn.2009-06.com.eucalyptus.euca-flacso-cluster1:store5,W407KmAb1yQLkoICLzOglSfB69dfaqVG2ZVYsmucRmPz4/+Uggkr1iAwk8VKkrGs3mW1USBDnzG02/0/ic8SYxnRINYhP2Oq2s16V0OiaMsQm4Mx2wXOUdwzjn50KEfqo8UYAslEj++1T0WLpAqMQAs3G7CDgi/UBuYG8LlJptqL7b19liIfgG9ifwO3T8YUxqda2JNnWKtLDs2z/CTrMBt0R6sMGGVUzOxw6Uk/yll/2mfFLk6bql6cdkT+zPDG7xGNcHs64Xu8FLiWjjnrwCNNKvZ53rqCUbqZ4AIdr5FtYQXepUPazqpRptRfZZdr2B6AkqExsdd9GGCCxJx9AA==,unknown,requested:/dev/vdb,attached } groupNames={ssh }
[Thu Jan 20 22:21:32 2011][012488][EUCADEBUG ] refresh_instances(): done
[Thu Jan 20 22:21:32 2011][012488][EUCADEBUG ] monitor_thread(): done

Thanks!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers