Description
===========
Hi, I am setting up multi node environment (controller, Nova, nuetron and cinder).
For now, I am able to create the blank Server and when I try to add or attach volume ( from cinder - volume from volume group) it fails.
Without volume I am able to launch machine.
Steps to reproduce
==================
1. Setup environment as Description area briefed
2. Try to build new instance
In this I don't know why it triggers tgtadm whereas I configured lvm as cinder.
sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.282 50710 DEBUG oslo_concurrency.processutils [req-70d6c927-9969-4dfb-b27e-94b7c95ed816 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - default default] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target" returned: 96 in 0.096s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:374
Error is below from logs:
========================
Failed to perform requested operation on instance "", the instance has an error status: Please try again later [Error: Build of instance aborted: Block Device Mapping is Invalid.].
Nova Compute Logs:
================
2017-04-22 14:31:48.517 14044 DEBUG nova.compute.claims [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] [instance: 79f65f4e-365c-45f7-808b-e9966de61681] Aborting claim: [Claim: 2048 MB memory, 20 GB disk] abort /usr/lib/python2.7/site-packages/nova/compute/claims.py:123
2017-04-22 14:31:48.517 14044 DEBUG oslo_concurrency.lockutils [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker.abort_instance_claim" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-04-22 14:31:48.521 14044 DEBUG oslo_messaging._drivers.amqpdriver [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] CALL msg_id: 97be26cf170c4df48cefab8812b7acbd exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
2017-04-22 14:31:48.594 14044 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 97be26cf170c4df48cefab8812b7acbd __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
2017-04-22 14:31:48.598 14044 DEBUG oslo_messaging._drivers.amqpdriver [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] CALL msg_id: 3c68a6ad11b44754be89faab6f65c98b exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
2017-04-22 14:31:48.617 14044 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 3c68a6ad11b44754be89faab6f65c98b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
2017-04-22 14:31:48.618 14044 DEBUG oslo_concurrency.lockutils [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.abort_instance_claim" :: held 0.101s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-04-22 14:31:48.619 14044 DEBUG nova.compute.manager [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] [instance: 79f65f4e-365c-45f7-808b-e9966de61681] Build of instance 79f65f4e-365c-45f7-808b-e9966de61681 aborted: Block Device Mapping is Invalid. _build_and_run_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:1937
2017-04-22 14:31:48.619 14044 WARNING nova.image.glance [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] No protocol specified in for api_server 'http://10.0.1.80:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)
2017-04-22 14:31:48.619 14044 DEBUG nova.compute.utils [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] [instance: 79f65f4e-365c-45f7-808b-e9966de61681] Build of instance 79f65f4e-365c-45f7-808b-e9966de61681 aborted: Block Device Mapping is Invalid. notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:313
2017-04-22 14:31:48.620 14044 ERROR nova.compute.manager [req-a1d191d7-d426-4f25-82ef-bf92663f3d18 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - - -] [instance: 79f65f4e-365c-45f7-808b-e9966de61681] Build of instance 79f65f4e-365c-45f7-808b-e9966de61681 aborted: Block Device Mapping is Invalid.
2017-04-22 14:31:48.620 14044 ERROR nova.compute.manager [instance: 79f65f4e-365c-45f7-808b-e9966de61681] Traceback (most recent call last):
2017-04-22 14:31:48.620 14044 ERROR nova.compute.manager [instance: 79f65f4e-365c-45f7-808b-e9966de61681] File
Cinder Messages
====================
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.187 50710 DEBUG oslo_concurrency.processutils [req-70d6c927-9969-4dfb-b27e-94b7c95ed816 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - default default] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.282 50710 DEBUG oslo_concurrency.processutils [req-70d6c927-9969-4dfb-b27e-94b7c95ed816 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - default default] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target" returned: 96 in 0.096s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:374
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.283 50710 DEBUG oslo_concurrency.processutils [req-70d6c927-9969-4dfb-b27e-94b7c95ed816 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - default default] u'sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:422
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server [req-70d6c927-9969-4dfb-b27e-94b7c95ed816 dc27f505a62c4181a0d03382fb99d744 67d52715de2244b1b32c9a34c914572d - default default] Exception during message handling
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4400, in initialize_connection
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server connector, volume=volume)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1414, in initialize_connection
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server volume, connector)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 833, in create_export
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server volume_path)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server **portals_config)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 137, in create_iscsi_target
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server run_as_root=True)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server Exit code: 96
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server Stdout: u''
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server Stderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\n'
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.rpc.server
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.287 50710 DEBUG oslo_messaging._drivers.amqpdriver [req-70d6c927-9969-4dfb-b27e-94b7c95ed816
Appears you are just missing the tgtadm utility on the node running cinder-volume :
Apr 22 14:31:42 storage cinder-volume: 2017-04-22 14:31:42.284 50710 ERROR oslo_messaging.