Activity log for bug #1779654

Date Who What changed Old value New value Message
2018-07-02 11:08:46 Eiji Kobayashi bug added bug
2018-07-02 13:03:20 Sean McGinnis tags drivers helion hpe
2018-07-18 06:37:07 Vivek Soni bug added subscriber Vivek Soni
2018-07-18 06:42:12 Vivek Soni bug added subscriber William Durairaj
2018-07-18 06:42:23 Vivek Soni bug added subscriber Imran Ansari
2018-08-01 06:25:06 Eiji Kobayashi description nova boot fails with "--max-count" option $ nova list +----+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +----+------+--------+------------+-------------+----------+ +----+------+--------+------------+-------------+----------+ $ nova boot --min-count=3 --max-count=3 --flavor iip_C01-M02 --block-device source=image,id=${MY_CENTOS72_IMG},dest=volume,size=10,shutdown=remove,bootindex=0 --key-name key-for-internal --security-groups sg-all-from-private-net --availability-zone AZ2 --nic net-id=${MY_WORK_NET} --nic net-id=${MY_PRV_NET} web_ext_az2 $ nova list +--------------------------------------+------------------------+--------+------------+-------------+----------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+----------------------------------------------------------+ | 168367d2-3a2e-4b62-b78f-f2c2e234c191 | web_ext_az2-1 | ACTIVE | - | Running | WORK-Net=192.168.30.21; Private-Net=192.168.10.24 | | 98173a45-0947-42ed-bcaa-e175ac8aeb6b | web_ext_az2-2 | ACTIVE | - | Running | WORK-Net=192.168.30.18; Private-Net=192.168.10.27 | | bbd9ada5-71e8-48a4-9a69-d36dbb5641a9 | web_ext_az2-3 | ERROR | - | NOSTATE | | +--------------------------------------+------------------------+--------+------------+-------------+----------------------------------------------------------+ <cinder-volume.log> 2018-06-07 08:03:48.787 17974 WARNING cinder.volume.manager [req-8a322777-ae06-4765-807c-d987f049288c 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (34a3e9a3-fca4-4ff5-8290-fd7c3e3528b7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' 2018-06-07 08:03:48.790 17974 WARNING cinder.volume.manager [req-8a322777-ae06-4765-807c-d987f049288c 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] Flow 'volume_create_manager' (074daf4a-94d5-4884-b420-6684e061b6a8) transitioned into state 'REVERTED' from state 'RUNNING' 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server [req-8a322777-ae06-4765-807c-d987f049288c 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] Exception during message handling 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 4367, in create_volume 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server allow_reschedule=allow_reschedule, volume=volume) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 635, in create_volume 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server _run_flow() 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 627, in _run_flow 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server flow_engine.run() 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server failures[0].reraise() 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 853, in execute 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server **volume_spec) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 790, in _create_from_image 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server image_service 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 669, in _create_from_image_download 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server image_service) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 550, in _copy_image_to_volume 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server raise exception.ImageCopyFailure(reason=ex) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Error (HTTP 500) OPERATION_FAILED - The operation failed: The volume list 'helion-cp1-c1-m1-mgmtVolumeList1528358624' cannot be created because a volume list with the same name already exists. Enter a different name for the new volume list. 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server The volume of cinder does not create if this issues occurred. If I set "hplefthand_debug = true" then following message recorded. 2018-06-07 08:45:57.440 30555 DEBUG cinder.volume.drivers.hpe.hpe_lefthand_iscsi [req-9bb0e9b4-33dc-4956-9a8b-baf8add719d5 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] <== initialize_connection: exception (247ms) VolumeBackendAPIException(u'Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "$1" already exists. Use a unique name and try again.',) trace_logging_wrapper /opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/utils.py:901 Increasing max-count specification increases the occurrence frequency. nova boot fails with "--max-count" option $ nova list +----+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +----+------+--------+------------+-------------+----------+ +----+------+--------+------------+-------------+----------+ $ nova boot --min-count=3 --max-count=3 --flavor XXflavor --block-device source=image,id=${MY_CENTOS_IMG},dest=volume,size=10,shutdown=remove,bootindex=0 --key-name key-for-internal --security-groups sg-all-from-private-net --availability-zone AZ2 --nic net-id=${MY_WORK_NET} --nic net-id=${MY_PRV_NET} web_ext_az2 $ nova list +--------------------------------------+------------------------+--------+------------+-------------+----------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+----------------------------------------------------------+ | 168367d2-3a2e-4b62-b78f-f2c2e234c191 | web_ext_az2-1 | ACTIVE | - | Running | WORK-Net=192.168.30.21; Private-Net=192.168.10.24 | | 98173a45-0947-42ed-bcaa-e175ac8aeb6b | web_ext_az2-2 | ACTIVE | - | Running | WORK-Net=192.168.30.18; Private-Net=192.168.10.27 | | bbd9ada5-71e8-48a4-9a69-d36dbb5641a9 | web_ext_az2-3 | ERROR | - | NOSTATE | | +--------------------------------------+------------------------+--------+------------+-------------+----------------------------------------------------------+ <cinder-volume.log> 2018-06-07 08:03:48.787 17974 WARNING cinder.volume.manager [req-8a322777-ae06-4765-807c-d987f049288c 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (34a3e9a3-fca4-4ff5-8290-fd7c3e3528b7) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' 2018-06-07 08:03:48.790 17974 WARNING cinder.volume.manager [req-8a322777-ae06-4765-807c-d987f049288c 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] Flow 'volume_create_manager' (074daf4a-94d5-4884-b420-6684e061b6a8) transitioned into state 'REVERTED' from state 'RUNNING' 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server [req-8a322777-ae06-4765-807c-d987f049288c 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] Exception during message handling 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 4367, in create_volume 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server allow_reschedule=allow_reschedule, volume=volume) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 635, in create_volume 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server _run_flow() 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/manager.py", line 627, in _run_flow 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server flow_engine.run() 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server failures[0].reraise() 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 853, in execute 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server **volume_spec) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 790, in _create_from_image 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server image_service 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 669, in _create_from_image_download 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server image_service) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server File "/opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 550, in _copy_image_to_volume 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server raise exception.ImageCopyFailure(reason=ex) 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Error (HTTP 500) OPERATION_FAILED - The operation failed: The volume list 'helion-cp1-c1-m1-mgmtVolumeList1528358624' cannot be created because a volume list with the same name already exists. Enter a different name for the new volume list. 2018-06-07 08:03:48.792 17974 ERROR oslo_messaging.rpc.server The volume of cinder does not create if this issues occurred. If I set "hplefthand_debug = true" then following message recorded. 2018-06-07 08:45:57.440 30555 DEBUG cinder.volume.drivers.hpe.hpe_lefthand_iscsi [req-9bb0e9b4-33dc-4956-9a8b-baf8add719d5 45336d44e7174de5b7d0430639437ccf 7ce30e9231de44009f10b555896105d4 - default default] <== initialize_connection: exception (247ms) VolumeBackendAPIException(u'Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "$1" already exists. Use a unique name and try again.',) trace_logging_wrapper /opt/stack/venv/cinder-20171028T231402Z/lib/python2.7/site-packages/cinder/utils.py:901 Increasing max-count specification increases the occurrence frequency.
2018-08-02 08:17:41 OpenStack Infra cinder: status New In Progress
2018-08-02 08:17:41 OpenStack Infra cinder: assignee Vivek Soni (viveksoni)
2018-08-02 17:34:54 OpenStack Infra cinder: status In Progress Fix Released
2018-08-09 01:50:47 OpenStack Infra tags drivers helion hpe drivers helion hpe in-stable-queens
2018-08-10 02:56:27 OpenStack Infra tags drivers helion hpe in-stable-queens drivers helion hpe in-stable-pike in-stable-queens
2018-08-20 18:59:17 OpenStack Infra tags drivers helion hpe in-stable-pike in-stable-queens drivers helion hpe in-stable-ocata in-stable-pike in-stable-queens
2018-08-24 16:18:40 OpenStack Infra tags drivers helion hpe in-stable-ocata in-stable-pike in-stable-queens drivers helion hpe in-driverfixes-newton in-stable-ocata in-stable-pike in-stable-queens