Unable to create the volume

[Vincent Hou]

This issue comes back, but the log is different. This is the c-vol log when I create a volume.

=====================================Log of c-vol=================================================
2012-09-28 13:33:43 DEBUG cinder.utils [req-f873bf75-7161-4c93-b090-39157d7ad965 17433a44527c48678968fd887c949c02 42514e875969444c9a89d8d12f18cfbb] Result was 1 from (pid=12720) execute /opt/stack/cinder/cinder/utils.py:180
2012-09-28 13:33:43 ERROR cinder.volume.iscsi [req-f873bf75-7161-4c93-b090-39157d7ad965 17433a44527c48678968fd887c949c02 42514e875969444c9a89d8d12f18cfbb] Failed to create iscsi target for volume id:volume-52041d8b-de5c-4877-9170-6c0849fb305c.
2012-09-28 13:33:43 ERROR cinder.openstack.common.rpc.amqp [-] Exception during message handling
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 163, in create_volume
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp volume_ref['id'], {'status': 'error'})
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 156, in create_volume
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_export(context, volume_ref)
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/driver.py", line 380, in create_export
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp volume_path)
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/iscsi.py", line 140, in create_iscsi_target
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-52041d8b-de5c-4877-9170-6c0849fb305c.
2012-09-28 13:33:43 TRACE cinder.openstack.common.rpc.amqp

[Vincent Hou (houshengbo)]

The VM seemed working fine, but the physical machine did not work.

[clayg (clay-gerrard)]

bit of a logging fail...

All of the interesting stuff is in the ProcessError, which is being eaten and translated to more specific but less informative ISCSITargetCreateFailed error. We get debug from utils line saying some sort of execute returned 1. From the code lines it looks like a tgt-admin --update

Is there another debug line above that says the sepecific tgt-admin command that was run?

[Vincent Hou (houshengbo)]

Finally I got the reason. Devstack has change $CINDER_STATE_PATH from /opt/stack/cinder/ to /opt/stack/data/cinder/. If there is no /etc/tgt/conf.d/cinder.conf existing, this config file will be created. However, if it already exists, its content will not be updated, so the path still remains the old one, which leads to the error.

[Rohit Karajgi (rohitk)]

Hi Vincent,

Can you please elaborate how you resolved this error?
The entry in the /etc/tgt/conf.d/cinder.conf on my setup was always
"include /opt/stack/data/cinder/volumes/*" and did not have to be updated.
I still see the this error but only on second cinder node.

[Everett Toews (everett-toews)]

I ran into this too. /etc/tgt/conf.d/cinder.conf was correct but still no love. Some combination of unstack.sh, stack.sh, and rejoin-stack.sh had killed my tgt daemon and failed to restart it so I had to restart it manually with

sudo /usr/sbin/service tgt restart

on Ubuntu 12.04

[Niknak (niknak-123)]

Same problem here. I followed the official instructions to setup openstack all in one with cinder on a fresh ubuntu 12.04 installation manually without any script.

http://docs.openstack.org/trunk/openstack-compute/install/apt/content/osfolubuntu-cinder.html

If i try to create a cinder volume on horizon i get an error message.

/var/log/cinder/cinder-volume.log shows the following entries:

2012-12-03 10:27:30 3897 DEBUG cinder.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'Member', u'admin'], u'_context_request_id': u'req-255fd640-910d-46f0-9ba9-9217c5e8b324', u'_context_quota_class': None, u'args': {u'image_id': None, u'snapshot_id': None, u'volume_id': u'f455afdb-93e7-4a66-8df5-257970dc9ce0'}, u'_context_auth_token': '<SANITIZED>', u'_context_is_admin': False, u'_context_project_id': u'08029e2c8da24363825fc266c6aebf09', u'_context_timestamp': u'2012-12-03T09:27:30.284264', u'_context_read_deleted': u'no', u'_context_user_id': u'6f960b2d844b48e8b9fd1d4212e6841d', u'method': u'create_volume', u'_context_remote_address': u'127.0.0.1'} _safe_log /usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/common.py:195
2012-12-03 10:27:30 3897 DEBUG cinder.openstack.common.rpc.amqp [-] unpacked context: {'user_id': u'6f960b2d844b48e8b9fd1d4212e6841d', 'roles': [u'Member', u'admin'], 'timestamp': u'2012-12-03T09:27:30.284264', 'auth_token': '<SANITIZED>', 'remote_address': u'127.0.0.1', 'quota_class': None, 'is_admin': False, 'request_id': u'req-255fd640-910d-46f0-9ba9-9217c5e8b324', 'project_id': u'08029e2c8da24363825fc266c6aebf09', 'read_deleted': u'no'} _safe_log /usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/common.py:195
2012-12-03 10:27:30 INFO cinder.volume.manager [req-255fd640-910d-46f0-9ba9-9217c5e8b324 6f960b2d844b48e8b9fd1d4212e6841d 08029e2c8da24363825fc266c6aebf09] volume volume-f455afdb-93e7-4a66-8df5-257970dc9ce0: creating
2012-12-03 10:27:30 DEBUG cinder.volume.manager [req-255fd640-910d-46f0-9ba9-9217c5e8b324 6f960b2d844b48e8b9fd1d4212e6841d 08029e2c8da24363825fc266c6aebf09] volume volume-f455afdb-93e7-4a66-8df5-257970dc9ce0: creating lv of size 1G create_volume /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:133
2012-12-03 10:27:30 DEBUG cinder.utils [req-255fd640-910d-46f0-9ba9-9217c5e8b324 6f960b2d844b48e8b9fd1d4212e6841d 08029e2c8da24363825fc266c6aebf09] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -L 1G -n volume-f455afdb-93e7-4a66-8df5-257970dc9ce0 cinder-volumes execute /usr/lib/python2.7/dist-packages/cinder/utils.py:156
2012-12-03 10:27:30 DEBUG cinder.volume.manager [req-255fd640-910d-46f0-9ba9-9217c5e8b324 6f960b2d844b48e8b9fd1d4212e6841d 08029e2c8da24363825fc266c6aebf09] volume volume-f455afdb-93e7-4a66-8df5-257970dc9ce0: creating export create_volume /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:155
2012-12-03 10:27:30 INFO cinder.volume.iscsi [req-255fd640-910d-46f0-9ba9-9217c5e8b324 6f960b2d844b48e8b9fd1d4212e6841d 08029e2c8da24363825fc266c6aebf09] Creating volume: volume-f455afdb-93e7-4a66-8df5-257970dc9ce0
2012-12-03 10:27:30 DEBUG cinder.utils [req-255fd640-910d-46f0-9ba9-9217c5e8b324 6f960b2d844b48e8b9fd1d4212e6841d 0802...

[Niknak (niknak-123)]

I figured it out. If i change the content of

/etc/tgt/targets.conf

to

#include /etc/tgt/conf.d/*.conf
include /etc/tgt/conf.d/cinder_tgt.conf
include /etc/tgt/conf.d/cinder.conf
include /etc/tgt/conf.d/nova_tgt.conf

it seems to works.

[cloud.abada (cloud-abada)]

Works perfectly for me too.

I changed:

include /etc/tgt/conf.d/*.conf
default-driver iscsi

to
include /etc/tgt/conf.d/cinder_tgt.conf
include /etc/tgt/conf.d/nova_tgt.conf
default-driver iscsi

Thanks NikNak.