Volume in error status after volume create

[Peng Peng]

Brief Description
-----------------
After volume create, volume status became "creating" and "downloading", eventually turns to "error" status

Severity
--------
Major

Steps to Reproduce
------------------
create volume
check volume status

TC-name: nova/test_migrate_vms.py::test_migrate_vm_various_guest[win_2016-4-4096-dedicated-volume]

Expected Behavior
------------------
status turns to available

Actual Behavior
----------------
status turns to error

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Two node system

Lab-name: IP_5-6

Branch/Pull Time/Commit
-----------------------
2020-05-31_20-00-00

Last Pass
---------
unknown

Timestamp/Logs
--------------
[2020-06-02 14:46:47,564] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --image=496b1cff-9e61-4737-9c96-aa5dbca80c94 --size=29 --bootable win_2016-5'
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-06-02T14:46:50.000000 |
| description | None |
| encrypted | False |
| id | 5af60ff4-813e-460a-955b-e3475daefa64 |
| multiattach | False |
| name | win_2016-5 |
| properties | |
| replication_status | None |
| size | 29 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | ceph-store |
| updated_at | None |
| user_id | 3dc2ccfc728a423fb77a1c5ae2ff51da |
+---------------------+--------------------------------------+

[2020-06-02 14:50:53,353] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume show 5af60ff4-813e-460a-955b-e3475daefa64'
[2020-06-02 14:50:55,255] 436 DEBUG MainThread ssh.expect :: Output:
+------------------------------+--------------------------------------+
| Field | Value |
+------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-06-02T14:46:50.000000 |
| description | None |
| encrypted | False |
| id | 5af60ff4-813e-460a-955b-e3475daefa64 |
| multiattach | False |
| name | win_2016-5 |
| os-vol-tenant-attr:tenant_id | c75e004bdd1f428891ebee51e58a6989 |
| properties | |
| replication_status | None |
| size | 29 |
| snapshot_id | None |
| source_volid | None |
| status | downloading |
| type | ceph-store |
| updated_at | 2020-06-02T14:49:55.000000 |
| user_id | 3dc2ccfc728a423fb77a1c5ae2ff51da |
| volume_image_metadata | {u'signature_verified': u'False'} |
+------------------------------+--------------------------------------+

[2020-06-02 14:56:59,677] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume show 5af60ff4-813e-460a-955b-e3475daefa64'
[2020-06-02 14:57:01,590] 436 DEBUG MainThread ssh.expect :: Output:
+------------------------------+--------------------------------------+
| Field | Value |
+------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-06-02T14:46:50.000000 |
| description | None |
| encrypted | False |
| id | 5af60ff4-813e-460a-955b-e3475daefa64 |
| multiattach | False |
| name | win_2016-5 |
| os-vol-tenant-attr:tenant_id | c75e004bdd1f428891ebee51e58a6989 |
| properties | |
| replication_status | None |
| size | 29 |
| snapshot_id | None |
| source_volid | None |
| status | error |
| type | ceph-store |
| updated_at | 2020-06-02T14:57:01.000000 |
| user_id | 3dc2ccfc728a423fb77a1c5ae2ff51da |
| volume_image_metadata | {u'signature_verified': u'False'} |
+------------------------------+--------------------------------------+

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

log added,
https://files.starlingx.kube.cengn.ca/launchpad/1881815

[Ghada Khalil (gkhalil)]

Assigning to Elena to check this new failure and determine if it is related to https://bugs.launchpad.net/starlingx/+bug/1819688 which she recently fixed. This issue is reported with a windows image

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - issue appears specific to windows images (which are typically very large)

[Elena Taivan (etaivan)]

The logs are not completed.

The volume was created at 14:46 on controller-1:
2020-06-02T14:46:47.000 controller-1 -sh: info HISTORY: PID=1216532 UID=42425 openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --image=496b1cff-9e61-4737-9c96-aa5dbca80c94 --size=29 --bootable win_2016-5

But there is no log in controller-1 in cinder-volume pod logs at that time.
The logs are starting at 14:55 and the only logs that I can see related to this volume are at 14:57
2020-06-02T14:57:01.181530194Z stdout F 2020-06-02 14:57:01.180 19 INFO cinder.manager [req-0ee55a2e-a3d5-44f9-bca0-ad196789cc3f - - - - -] Cleaning Volume with id 5af60ff4-813e-460a-955b-e3475daefa64 and status downloading

I'm not able to see what's happening when the volume was created.

What can I tell for sure is that 'image-conversion' filesystem was not added. I checked in bash.log file and I don't see any 'system host-fs-add controller image-conversion' command.

[zhipeng liu (zhipengs)]

Hi Pengpeng,

Could you help to retest this case with creating 'image-conversion' filesystem since large size image
creation has a known issue that we fixed.

Thanks!
Zhipeng

[Elena Taivan (etaivan)]

Please check my latest comments added here : https://jira.windriver.com/browse/CGTS-18695 .
Please use image-conversion filesystem.

[Elena Taivan (etaivan)]

To add image-conversion, please look at this doc review with the instructions https://review.opendev.org/#/c/730529/2/doc/source/configuration/openstack_cinder_config.rst