This issue is only for NFS driver.
If user tries to boot instance from volume which was earlier used for booting instance throws permission denied error for executing
command "/usr/bin/python m oslo_concurrency.prlimit --as=1073741824 --cpu=8 - env LC_ALL=C qemu-img info /opt/stack/data/cinder/mnt/431c1833ed03dbc054f2ff7752ef9953/volume-4c729508-4b9b-4201-b805-e134e5fdb032" at cinder side.
Steps to reproduce:
1. Create bootable volume.
$ cinder create --image <image-uuid> --name bootable 1
2. Boot instance using created volume.
$ nova boot --boot-volume <volume-uuid> --flavor <flavor-id> boot-from-volume
3. Once instance becomes active, delete the instance.
nova delete <instance-uuid>
4. After deleting the instance, volume becomes available. Use this volume to boot another instance.
$ nova boot --boot-volume <volume-uuid> --flavor <flavor-id> boot-from-volume
This operation fails and instance goes into ERROR state with following logs in nova compute and volume node.
ERROR nova.compute.manager [[01;36mreq-6447840e-e913-477c-9d15-f15c86dbe638 [00;36mdemo demo] [01;35m[instance: 8322d1da-28db-4336-8691-886592a9468b] Build of instance 8322d1da-28db-4336-8691-886592a9468b aborted: Block Device Mapping is Invalid.[00m
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 1775, in _do_build_and_run_instance
filter_properties)
File "/opt/stack/nova/nova/compute/manager.py", line 1956, in _build_and_run_instance
phase=fields.NotificationPhase.ERROR, exception=e)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 1909, in _build_and_run_instance
block_device_mapping) as resources:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/opt/stack/nova/nova/compute/manager.py", line 2108, in _build_resources
reason=e.format_message())
BuildAbortException: Build of instance 8322d1da-28db-4336-8691-886592a9468b aborted: Block Device Mapping is Invalid.
ERROR nova.compute.manager [[01;36mreq-6447840e-e913-477c-9d15-f15c86dbe638 [00;36mdemo demo] [01;35m[instance: 8322d1da-28db-4336-8691-886592a9468b] Instance failed block device setup[00m
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 1579, in _prep_block_device
wait_func=self._await_block_device_map_created)
File "/opt/stack/nova/nova/virt/block_device.py", line 620, in attach_block_devices
_log_and_attach(device)
File "/opt/stack/nova/nova/virt/block_device.py", line 617, in _log_and_attach
bdm.attach(*attach_args, **attach_kwargs)
File "/opt/stack/nova/nova/virt/block_device.py", line 48, in wrapped
ret_val = method(obj, context, *args, **kwargs)
File "/opt/stack/nova/nova/virt/block_device.py", line 369, in attach
connector)
File "/opt/stack/nova/nova/volume/cinder.py", line 177, in wrapper
res = method(self, ctx, *args, **kwargs)
File "/opt/stack/nova/nova/volume/cinder.py", line 199, in wrapper
res = method(self, ctx, volume_id, *args, **kwargs)
File "/opt/stack/nova/nova/volume/cinder.py", line 383, in initialize_connection
exc.code if hasattr(exc, 'code') else None)})
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/volume/cinder.py", line 357, in initialize_connection
context).volumes.initialize_connection(volume_id, connector)
File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 404, in initialize_connection
{'connector': connector})
File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 334, in _action
resp, body = self.api.client.post(url, body=body)
File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 174, in post
return self._cs_request(url, 'POST', **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 162, in _cs_request
return self.request(url, method, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 151, in request
raise exceptions.from_response(resp, body)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-685539f3-56a2-4df1-bef2-d4dd43b6e64e)
Cinder Volume error logs:
2017-04-12 01:25:06.099 ERROR cinder.volume.manager [[01;36mreq-230ed749-9d51-4a9b-9400-cf8e9bd6dab2 [00;36mdemo None] [01;35mDriver initialize connection failed (error: Unexpected error while running command.
Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=8 -- env LC_ALL=C qemu-img info /opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b
Exit code: 1
Stdout: u''
Stderr: u"qemu-img: Could not open '/opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b': Could not open '/opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b': Permission denied\n").[00m
Traceback (most recent call last):
File "/opt/stack/cinder/cinder/volume/manager.py", line 1510, in initialize_connection
conn_info = self.driver.initialize_connection(volume, connector)
File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 131, in initialize_connection
info = self._qemu_img_info(path_to_vol, volume['name'])
File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 534, in _qemu_img_info
path, volume_name, self.configuration.nfs_mount_point_base)
File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 707, in _qemu_img_info_base
run_as_root=self._execute_as_root)
File "/opt/stack/cinder/cinder/image/image_utils.py", line 82, in qemu_img_info
prlimit=QEMU_IMG_LIMITS)
File "/opt/stack/cinder/cinder/utils.py", line 123, in execute
return processutils.execute(*cmd, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 400, in execute
cmd=sanitized_cmd)
ProcessExecutionError: Unexpected error while running command.
Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=8 -- env LC_ALL=C qemu-img info /opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b
Exit code: 1
Stdout: u''
Stderr: u"qemu-img: Could not open '/opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b': Could not open '/opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b': Permission denied\n"
[01;36mreq-230ed749-9d51-4a9b-9400-cf8e9bd6dab2 [00;36mdemo None] [01;35mException during message handling[00m
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/cinder/cinder/volume/manager.py", line 1518, in initialize_connection
raise exception.VolumeBackendAPIException(data=err_msg)
VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Unexpected error while running command.
Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=8 -- env LC_ALL=C qemu-img info /opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b
Exit code: 1
Stdout: u''
Stderr: u"qemu-img: Could not open '/opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b': Could not open '/opt/stack/data/cinder/mnt/94e676b56dfb1a3d00d7d35830c1a1ce/volume-df8a7570-ab3b-48c3-b78b-b454a360784b': Permission denied\n").
Hi Abhishek,
I tried to reproduce the issue but my second instance created successfully and in active state.
can you provide more details about reporducing the same ?