Attaching a volume to a "pending" instance claims success, but never attaches.

[Kiall Mac Innes]

Attaching a volume to a "pending" instance claims success (via euca2ools, and the API) but never attaches to the instance. This may be a KVM only issue.

As its pretty easy to reproduce, I haven't attached much in the way of detail - let me know if you need any specific info.

Command Line usage, assuming i-307905AD is in the "pending" state:

$ euca-attach-volume -d /dev/vda -i i-307905AD vol-59820625
VOLUME vol-59820625

Also, this possibly only occurs very early (first 5 seconds) of the pending state. The same command succeeds if you wait for the instance to arrive at the "running" status.

Eucalyptus: Version: 2.0+bzr1241-0ubuntu4
Euca2ools: Version: 1.2-0ubuntu11

[Luca Invernizzi (invernizzi)]

This seems to be due to a incompatibility between eucalyptus-nc and libvirt

When attaching a disk (/dev/sdX or /dev/vdX), this is the output on the node controller:

[Thu Nov 24 23:33:56 2011][000571][EUCAINFO ] doAttachVolume() invoked (id=i-416607C2 vol=vol-5A24063B remote=//,192.35.222.25,iqn.2009-06.com.eucalyptus.secluster:store3,djsFVOsygc3D8GA6SAYtxMQRPLFlzdkXXPakJKalNo3lE8i9GKasvANekX4fETg3RSAv7zQgoRyQFOjc/vfnb8tqjdb9jJsInIzHuYHcDkihfXrYiSGmGxXzh2ijjpPb1BYAQZI2OwSM2Pux+0lwNJwGxl7zW2pVeSpFT9pulZahLH7eX/RcXaHWl/glKqMtxz/L3g06OlM/bneoFofCGZiPxBklDTiC2uz/HN3Ew+JF5u/PePv5qAFVlv61jJk4f80G7ZMM7+LZfjSmi4TVm/xAlRLzqiQbkmvabpXVY9utCBQS2qm46bFeZMFXaxf14iIIsDX3K0XD9SkOUAA5KQ== local=/dev/vda)
[Thu Nov 24 23:33:56 2011][000571][EUCAINFO ] connect_iscsi_target invoked (dev_string=//,192.35.222.25,iqn.2009-06.com.eucalyptus.secluster:store3,djsFVOsygc3D8GA6SAYtxMQRPLFlzdkXXPakJKalNo3lE8i9GKasvANekX4fETg3RSAv7zQgoRyQFOjc/vfnb8tqjdb9jJsInIzHuYHcDkihfXrYiSGmGxXzh2ijjpPb1BYAQZI2OwSM2Pux+0lwNJwGxl7zW2pVeSpFT9pulZahLH7eX/RcXaHWl/glKqMtxz/L3g06OlM/bneoFofCGZiPxBklDTiC2uz/HN3Ew+JF5u/PePv5qAFVlv61jJk4f80G7ZMM7+LZfjSmi4TVm/xAlRLzqiQbkmvabpXVY9utCBQS2qm46bFeZMFXaxf14iIIsDX3K0XD9SkOUAA5KQ==)
[Thu Nov 24 23:33:56 2011][000571][EUCADEBUG ] system_output(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/connect_iscsitarget.pl //,192.35.222.25,iqn.2009-06.com.eucalyptus.secluster:store3,djsFVOsygc3D8GA6SAYtxMQRPLFlzdkXXPakJKalNo3lE8i9GKasvANekX4fETg3RSAv7zQgoRyQFOjc/vfnb8tqjdb9jJsInIzHuYHcDkihfXrYiSGmGxXzh2ijjpPb1BYAQZI2OwSM2Pux+0lwNJwGxl7zW2pVeSpFT9pulZahLH7eX/RcXaHWl/glKqMtxz/L3g06OlM/bneoFofCGZiPxBklDTiC2uz/HN3Ew+JF5u/PePv5qAFVlv61jJk4f80G7ZMM7+LZfjSmi4TVm/xAlRLzqiQbkmvabpXVY9utCBQS2qm46bFeZMFXaxf14iIIsDX3K0XD9SkOUAA5KQ==]
[Thu Nov 24 23:33:57 2011][000571][EUCAINFO ] Attached device: /dev/sdc
[Thu Nov 24 23:33:57 2011][000571][EUCAERROR ] libvirt: internal error unsupported driver name 'phy' for disk '/dev/sdc' (code=1)
[Thu Nov 24 23:33:57 2011][000571][EUCAERROR ] virDomainAttachDevice() failed (err=-1) XML=<disk type='block'><driver name='phy'/><source dev='/dev/sdc'/><target dev='vda'/></disk>
[Thu Nov 24 23:33:57 2011][000571][EUCAERROR ] ERROR: doAttachVolume() failed error=1

Please note the line: libvirt: internal error unsupported driver name 'phy' for disk '/dev/sdc' .
The driver name should be 'quemu'. Libvirt used to tolerate 'phy', but it doesn't anymore (as in https://bugzilla.redhat.com/show_bug.cgi?id=627143#c13).
I haven't had time to try it yet, but it appears that substituting "phy" with "quemu" in http://bazaar.launchpad.net/~eucalyptus-maintainers/eucalyptus/2.0.0-lp/view/head:/node/handlers_kvm.c would solve the problem.

[Luca Invernizzi (invernizzi)]

Obviously, I meant "qemu".

[Luca Invernizzi (invernizzi)]

As I suspected, the incorrect disk driver for kvm was the issue.
I've attached a patch in the debian patch format, that lets me successfully attach disks.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "31-handlers_kvm.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Dmitrii Zagorodnov (dmitrii)]

Luca - thanks for the patch! Looks like your solution is the right one for KVM. I'd like to ensure that volume attachments continue to work on Xen (driver name 'phy' is Xen-specific, which used to be ignored by QEMU until recently). So a bit of extra logic may be necessary to differentiate XML generated for Xen and KVM. I'll test and will get back to you.

[Dmitrii Zagorodnov (dmitrii)]

Ah, my bad, since it's a patch to handlers_kvm.c, Xen code path is not affected. I've confirmed that 'qemu' setting works with older Ubuntu releases, too. So, we'll be happy to take this as is.

Luca - are you OK with signing off the ownership of the patch to us?

[Luca Invernizzi (invernizzi)]

Sure.
Thanks for your help

[graziano obertelli (graziano.obertelli)]

Thanks Luca: assigning it to milestone 2.0.4.

[Andy Grimm (agrimm)]

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2742

Please watch that issue for further updates.