fail to destroy an instance launched from gluster volume when storage is inaccessible

[Dafna Ron]

I am running cinder with gluster backend.
at some point my gluster has crashed completely and when I try to destroy the instance I cannot:

2013-10-09 16:43:28.228 7297 ERROR cinder.openstack.common.rpc.common [req-6c4719af-0e90-47f6-b60d-5f17f4d0411e c02995f25ba44cfab1a3cbd419f045a1 c77235c29fd0431a8e6628ef6d18e07f] ['Traceback (most recent call last):\n', ' File "/usr/lib
/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data\n **args)\n', ' File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch\n return getattr(pr
oxyobj, method)(ctxt, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 502, in detach_volume\n self.driver.ensure_export(context, volume)\n', ' File "/usr/lib/python2.6/site-packages/cinder/volu
me/drivers/glusterfs.py", line 816, in ensure_export\n self._ensure_share_mounted(volume[\'provider_location\'])\n', ' File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 973, in _ensure_share_mounted\n
  self._mount_glusterfs(glusterfs_share, mount_path, ensure=True)\n', ' File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 1041, in _mount_glusterfs\n self._execute(\'mkdir\', \'-p\', mount_path)\n', '
File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute\n
    cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: mkdir -p /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364\nExit code: 1\nStdout: \'\'\nStderr: "mkdir: cannot create directory `/v
ar/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364\': File exists\\n"\n']
2013-10-09 16:43:32.434 7297 ERROR cinder.openstack.common.rpc.amqp [req-c5d059b9-75f7-4151-ab30-1287817f7800 c02995f25ba44cfab1a3cbd419f045a1 c77235c29fd0431a8e6628ef6d18e07f] Exception during message handling
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp **args)
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 502, in detach_volume
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp self.driver.ensure_export(context, volume)
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 816, in ensure_export
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp self._ensure_share_mounted(volume['provider_location'])
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 973, in _ensure_share_mounted
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp self._mount_glusterfs(glusterfs_share, mount_path, ensure=True)
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 1041, in _mount_glusterfs
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp self._execute('mkdir', '-p', mount_path)
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp return processutils.execute(*cmd, **kwargs)
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp cmd=' '.join(cmd))
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp Command: mkdir -p /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp Stdout: ''
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp Stderr: "mkdir: cannot create directory `/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364': File exists\n"
2013-10-09 16:43:32.434 7297 TRACE cinder.openstack.common.rpc.amqp
2013-10-09 16:43:32.436 7297 ERROR cinder.openstack.common.rpc.common [req-c5d059b9-75f7-4151-ab30-1287817f7800 c02995f25ba44cfab1a3cbd419f045a1 c77235c29fd0431a8e6628ef6d18e07f] Returning exception Unexpected error while running command
.
Command: mkdir -p /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364
Exit code: 1
Stdout: ''
Stderr: "mkdir: cannot create directory `/var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364': File exists\n" to caller

the instance no longer exists in libvirt:

root@cougar06 ~(keystone_admin)]# virsh -r list
 Id Name State
----------------------------------------------------

[root@cougar06 ~(keystone_admin)]#

but nova shows it:

root@cougar06 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| e7d99f1b-518f-4742-9d98-ab3053c4c806 | test | ACTIVE | deleting | NOSTATE | novanetwork=192.168.32.2 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
[root@cougar06 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| e7d99f1b-518f-4742-9d98-ab3053c4c806 | test | ACTIVE | None | NOSTATE | novanetwork=192.168.32.2 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
[root@cougar06 ~(keystone_admin)]# virsh -r list
 Id Name State
----------------------------------------------------

[root@cougar06 ~(keystone_admin)]# nova show e7d99f1b-518f-4742-9d98-ab3053c4c806
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-10-09T15:32:59Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | XXXX.XXl.XX.XX.redhat.com |
| key_name | None |
| image | Attempt to boot from volume - no image supplied |
| hostId | 42aee086631922deee420d708c6833a79caf713221604e1d5883d194 |
| OS-EXT-STS:vm_state | active |
| OS-EXT-SRV-ATTR:instance_name | instance-0000000d |
| OS-SRV-USG:launched_at | 2013-10-08T16:52:42.000000 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cougar06.scl.lab.tlv.redhat.com |
| flavor | m1.tiny (1) |
| id | e7d99f1b-518f-4742-9d98-ab3053c4c806 |
| security_groups | [{u'name': u'default'}] |
| OS-SRV-USG:terminated_at | None |
| user_id | c02995f25ba44cfab1a3cbd419f045a1 |
| name | test |
| created | 2013-10-08T16:52:34Z |
| tenant_id | c77235c29fd0431a8e6628ef6d18e07f |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| novanetwork network | 192.168.32.2 |
| os-extended-volumes:volumes_attached | [{u'id': u'e78978af-0f46-4caf-948b-218afb5de6ef'}] |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+--------------------------------------+----------------------------------------------------------+

It might be connected to the fact that we do not set timeout in nfsmount.conf and the mount/link is actually still there:

XXXXX:/Dafna_glance on /var/lib/glance/images type fuse.glusterfs (rw,default_permissions,default_permissions,allow_other,max_read=131072,allow_other,max_read=131072)
XXXXX:/Dafna_rhos on /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
XXXXX:/Dafna_rhos on /var/lib/nova/mnt/4a31bc6e5fb9244971075aa23d364364 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

root@cougar06 ~(keystone_admin)]# ls -l /var/lib/nova/mnt/
ls: cannot access /var/lib/nova/mnt/4a31bc6e5fb9244971075aa23d364364: Transport endpoint is not connected
total 0
d????????? ? ? ? ? ? 4a31bc6e5fb9244971075aa23d364364
[root@cougar06 ~(keystone_admin)]# ls -l /var/lib/cinder/mnt/
ls: cannot access /var/lib/cinder/mnt/4a31bc6e5fb9244971075aa23d364364: Transport endpoint is not connected
total 0
d????????? ? ? ? ? ? 4a31bc6e5fb9244971075aa23d364364
[root@cougar06 ~(keystone_admin)]# ls -l /var/lib/glance/
ls: cannot access /var/lib/glance/images: Transport endpoint is not connected
total 0
d????????? ? ? ? ? ? images
[root@cougar06 ~(keystone_admin)]#

[John Griffith (john-griffith)]

Marking incomplete due to no activity for almost a year