[2.0+bzr1241-0ubuntu4.1] Unstable state for the iscsi daemon

[Razique Mahroua]

Description: Ubuntu 10.10
Release: 10.10
UEC suite : 2.0+bzr1241-0ubuntu4.1

Description :
After doing tests of custom EMI's instancitation, the Node Controller went unstable. I was then unable to instanciate anymore instances. For the running instances with EBS attached, I was then unable to detach them, and for new instances, unable to attach a new EBS volume :

Detaching :

[Wed May 4 11:01:23 2011][020884][EUCAERROR ] ERROR: DetachVolume returned an error
[Wed May 4 11:01:23 2011] ERROR: doDetachVolume() returned FAIL

Attaching :
[Wed May 4 09:26:04 2011][013319][EUCAERROR ] libvirt: internal error '//,46.227.16.70,iqn.2009-06.com.eucalyptus.cluster1:store5,Itj/2Zbs1QNRlVzKROjF7k5nKBnck0NuFPlgm2hlf7F63sMKldO0W3vG4at6RE4PRe5gy6tKMPnkpZXAUQlymBOT+hAUmT4cBgvDYNdmAN98oLlUGXqkzZVI3SxBBfUeHwAi7v/tQHcZUg49ezVWpdXg4D55DUHT6BwsxLYmxXmbqRUH5RVeXw7r8Dhaplh1P5X5KWZHvdCrQDWbUF8zU37jxOsUANzzpTYUTfRl7VtHitvzQ2ebY06xRbABziBe6MY/p3r6qILyl8vDgot1MQ+OzWH3WizsDRu4g0uDWI3MVfKUh2UEz2mOA/IKSQF7Cfrv2zyIpUSGq+uDPhp/hg==' does not exist (code=1)

In order to be able to instanciate, I manually restarted the iscsi part :
$ service open-iscsi restart

which cleaned all the running sessions, but lead to :
[Wed May 4 09:34:32 2011][013319][EUCAERROR ] libvirt: operation failed: disk sde not found (code=9)
when I was trying to dettach disks.

Even by restarting all the services (NC, NC-publication, SC, SC-publication), UEC wouldn't clean the missing volumes. UEC kept in memory iscsi sessions and attachments (on the node controller (/dev/sdX and when euca2ools-describe-volumes invoked, the volumes were still present))

Expected behaviour :
- When all the components are restarted, the EBS sessions should be cleaned

What happened :
- Unable to get rid of "Attaching..." or "Dettaching..." status, due to missing iscsi volumes

Razique

[Andy Grimm (agrimm)]

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2767

Please watch that issue for further updates.