[UEC 2.0+bzr1241-0ubuntu4.1] Unstable state for the iscsi daemon

[Razique Mahroua]

Description: Ubuntu 10.10
Release: 10.10
UEC suite : 2.0+bzr1241-0ubuntu4.1

Description :
After doing tests of custom EMI's instancitation, the Node Controller went unstable. I was then unable to instanciate anymore instances. For the running instances with EBS attached, I was then unable to detach them, and for new instances, unable to attach a new EBS volume :

Detaching :

[Wed May 4 11:01:23 2011][020884][EUCAERROR ] ERROR: DetachVolume returned an error
[Wed May 4 11:01:23 2011] ERROR: doDetachVolume() returned FAIL

Attaching :
[Wed May 4 09:26:04 2011][013319][EUCAERROR ] libvirt: internal error '//,46.227.16.70,iqn.2009-06.com.eucalyptus.cluster1:store5,Itj/2Zbs1QNRlVzKROjF7k5nKBnck0NuFPlgm2hlf7F63sMKldO0W3vG4at6RE4PRe5gy6tKMPnkpZXAUQlymBOT+hAUmT4cBgvDYNdmAN98oLlUGXqkzZVI3SxBBfUeHwAi7v/tQHcZUg49ezVWpdXg4D55DUHT6BwsxLYmxXmbqRUH5RVeXw7r8Dhaplh1P5X5KWZHvdCrQDWbUF8zU37jxOsUANzzpTYUTfRl7VtHitvzQ2ebY06xRbABziBe6MY/p3r6qILyl8vDgot1MQ+OzWH3WizsDRu4g0uDWI3MVfKUh2UEz2mOA/IKSQF7Cfrv2zyIpUSGq+uDPhp/hg==' does not exist (code=1)

In order to be able to instanciate, I manually restarted the iscsi part :
$ service open-iscsi restart

which cleaned all the running sessions, but lead to :
[Wed May 4 09:34:32 2011][013319][EUCAERROR ] libvirt: operation failed: disk sde not found (code=9)
when I was trying to dettach disks.

Even by restarting all the services (NC, NC-publication, SC, SC-publication), UEC wouldn't clean the missing volumes. UEC kept in memory iscsi sessions and attachments (on the node controller (/dev/sdX and when euca2ools-describe-volumes invoked, the volumes were still present))

Expected behaviour :
- When all the components are restarted, the EBS sessions should be cleaned

What happened :
- Unable to get rid of "Attaching..." or "Dettaching..." status, due to missing iscsi volumes

Razique

[Dave Walker (davewalker)]

This issue needs to be reproduced in order to be confirmed.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Edmund Haselwanter (ehaselwanter)]

Possible in a related state.

2012-06-18 08:59:13 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2012-06-18 08:59:13 TRACE nova.rpc.amqp Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000030 -p 192.168.124.87:3260 --op update -n node.startup -v manual
2012-06-18 08:59:13 TRACE nova.rpc.amqp Exit code: 255
2012-06-18 08:59:13 TRACE nova.rpc.amqp Stdout: ''
2012-06-18 08:59:13 TRACE nova.rpc.amqp Stderr: 'iscsiadm: no records found!\n'
2012-06-18 08:59:13 TRACE nova.rpc.amqp

root cause is a successful detach of the volume followed by another attempt after this successful detach (we try to create a reproduce able set of commands to get into this state)