HTTPS: nova.exception.CinderConnectionFailed' when boot VM from volume ERROR (ClientException): Unexpected API Error

[Anujeyan Manokeran]

Brief Description
-----------------.
     Below VM launch command failed with exception with unexpected API Error.
nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --boot-volume=6a47ae76-a2d1-4db6-bde6-44622c83809c --flavor=24f9af18-435c-4198-9ee0-29c15a92d4eb --poll --key-name=keypair-tenant1 --nic net-id=d3602b78-6c37-4a67-84de-7b9efdbe072b --nic port-id=e2c0bea7-e127-4d65-a055-2476a095eb25 --nic port-id=9e8d5f81-c1f9-4dbc-aa5f-bfa7859bf6ca tenant1-virtio-0-1'

system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-08-25 00:34:46,689] 423 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-17-centos-stable-versioned | armada-manifest | stx-openstack.yaml | applied | completed |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+

nd 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list'
[2019-08-25 00:34:50,226] 423 DEBUG MainThread ssh.expect :: Output:
+----+---------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+-----------------+-------+
| 4 | compute-0 | QEMU | 192.168.206.149 | up |
| 7 | compute-1 | QEMU | 192.168.206.188 | up |
+----+---------------------+-----------------+-----------------+-------+
'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne quota show tenant1'
[2019-08-25 00:34:52,254] 423 DEBUG MainThread ssh.expect :: Output:
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| backup-gigabytes | 1000 |
| backups | 10 |
| cores | 9 |
| fixed-ips | -1 |
| floating-ips | 3 |
| gigabytes | 1000 |
| gigabytes_ceph-store | -1 |
| groups | 10 |
| health_monitors | None |
| injected-file-size | 10240 |
| injected-files | 5 |
| injected-path-size | 255 |
| instances | 3 |
| key-pairs | 100 |
| l7_policies | None |
| listeners | None |
| load_balancers | None |
| location | Munch({'project': Munch({'domain_name': 'Default', 'domain_id': None, 'name': 'admin', 'id': u'66bc5c21e1f340118e53b00ecb762939'}), 'cloud': '', 'region_name': u'RegionOne', 'zone': None}) |
| name | None |
| networks | 5 |
| per-volume-gigabytes | -1 |
| pools | None |
| ports | 51 |
| project | 8eb5c4c9e2f54917bf3fa72a2a3e4123 |
| project_name | tenant1 |
| properties | 128 |
| ram | 51200 |
| rbac_policies | 10 |
| routers | 10 |
| secgroup-rules | 100 |
| secgroups | 10 |
| server-group-members | 10 |
| server-groups | 10 |
| snapshots | 6 |
| snapshots_ceph-store | -1 |
| subnet_pools | -1 |
| subnets | 15 |
| volumes | 6 |
| volumes_ceph-store | -1 |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Cinder volume is created and available
'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume show 6a47ae76-a2d1-4db6-bde6-44622c83809c'
[2019-08-25 00:37:38,148] 423 DEBUG MainThread ssh.expect :: Output:
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2019-08-25T00:37:36.000000 |
| description | None |
| encrypted | False |
| id | 6a47ae76-a2d1-4db6-bde6-44622c83809c |
| multiattach | False |
| name | vol-tenant1-virtio-0-1-1 |
| os-vol-tenant-attr:tenant_id | 8eb5c4c9e2f54917bf3fa72a2a3e4123 |
| properties | |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | ceph-store |
| updated_at | 2019-08-25T00:37:36.000000 |
| user_id | a8d7887854194f92b93cdbdf1559f89e |
| volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'raw', u'image_name': u'tis-centos-guest', u'image_id': u'12262e85-84f7-498b-bcad-7f09fbd1b76d', u'checksum': u'b0e308015909fd2ac4c7472e19f149de', u'min_disk': u'0', u'size': u'1611581440'} |
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[2019-08-25 00:37:38,252] 301 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --boot-volume=6a47ae76-a2d1-4db6-bde6-44622c83809c --flavor=24f9af18-435c-4198-9ee0-29c15a92d4eb --poll --key-name=keypair-tenant1 --nic net-id=d3602b78-6c37-4a67-84de-7b9efdbe072b --nic port-id=e2c0bea7-e127-4d65-a055-2476a095eb25 --nic port-id=9e8d5f81-c1f9-4dbc-aa5f-bfa7859bf6ca tenant1-virtio-0-1'
[2019-08-25 00:37:39,985] 423 DEBUG MainThread ssh.expect :: Output:
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.CinderConnectionFailed'> (HTTP 500) (Request-ID: req-6796326e-f489-4d40-b21c-9d224133480f)
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-08-25 00:37:39,985] 301 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-08-25 00:37:40,088] 423 DEBUG MainThread ssh.expect :: Output:
1

Severity
--------
Major

Steps to Reproduce
------------------
1. Check system health make sure healthy
2. Create flavor
'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor create --vcpus=2 --ram=1024 --disk=2 virtio'
[2019-08-25 00:37:20,310] 423 DEBUG MainThread ssh.expect :: Output:
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 2 |
| id | 24f9af18-435c-4198-9ee0-29c15a92d4eb |
| name | virtio |
| os-flavor-access:is_public | True |
| properties | |
| ram | 1024 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+----------------------------+--------------------------------------+
3. Create Network virtio-1
openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne port create --network=064a2ec7-2e0e-4dd6-8c7f-b774c1c844c3 --vnic-type=normal port_virtio-1'
[2019-08-25 00:37:28,321] 423 DEBUG MainThread ssh.expect :: Output:
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | None |
| binding_profile | None |
| binding_vif_details | None |
| binding_vif_type | None |
| binding_vnic_type | normal |
| created_at | 2019-08-25T00:37:27Z |
| data_plane_status | None |
| description | |
| device_id | |
| device_owner | |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='172.16.2.129', subnet_id='02728ba6-4900-46ae-ae63-ae4cc9eb72fb' |
| id | e2c0bea7-e127-4d65-a055-2476a095eb25 |
| location | Munch({'project': Munch({'domain_name': 'Default', 'domain_id': None, 'name': 'tenant1', 'id': u'8eb5c4c9e2f54917bf3fa72a2a3e4123'}), 'cloud': '', 'region_name': u'RegionOne', 'zone': None}) |
| mac_address | fa:16:3e:71:cb:55 |
| name | port_virtio-1 |
| network_id | 064a2ec7-2e0e-4dd6-8c7f-b774c1c844c3 |
| port_security_enabled | True |
| project_id | 8eb5c4c9e2f54917bf3fa72a2a3e4123 |
| propagate_uplink_status | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 1 |
| security_group_ids | e4d18b60-b043-437a-a2ac-2e29705217a5 |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2019-08-25T00:37:27Z |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4. Create Cinder volume
'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --bootable --image=12262e85-84f7-498b-bcad-7f09fbd1b76d --size=2 vol-tenant1-virtio-0-1-1'
[2019-08-25 00:37:36,429] 423 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-25T00:37:36.000000 |
| description | None |
| encrypted | False |
| id | 6a47ae76-a2d1-4db6-bde6-44622c83809c |
| multiattach | False |
| name | vol-tenant1-virtio-0-1-1 |
| properties | |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | ceph-store |
| updated_at | None |
| user_id | a8d7887854194f92b93cdbdf1559f89e |
+---------------------+--------------------------------------+

5. Volume created . Vm creation failure with exception as below
'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume show 6a47ae76-a2d1-4db6-bde6-44622c83809c'
[2019-08-25 00:37:38,148] 423 DEBUG MainThread ssh.expect :: Output:
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2019-08-25T00:37:36.000000 |
| description | None |
| encrypted | False |
| id | 6a47ae76-a2d1-4db6-bde6-44622c83809c |
| multiattach | False |
| name | vol-tenant1-virtio-0-1-1 |
| os-vol-tenant-attr:tenant_id | 8eb5c4c9e2f54917bf3fa72a2a3e4123 |
| properties | |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | ceph-store |
| updated_at | 2019-08-25T00:37:36.000000 |
| user_id | a8d7887854194f92b93cdbdf1559f89e |
| volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'raw', u'image_name': u'tis-centos-guest', u'image_id': u'12262e85-84f7-498b-bcad-7f09fbd1b76d', u'checksum': u'b0e308015909fd2ac4c7472e19f149de', u'min_disk': u'0', u'size': u'1611581440'} |
+------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-08-25 00:37:38,148] 301 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-08-25 00:37:38,251] 423 DEBUG MainThread ssh.expect :: Output:
0
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-08-25 00:37:38,252] 406 INFO MainThread cinder_helper.__wait_for_vol_status:: Volume 6a47ae76-a2d1-4db6-bde6-44622c83809c is in available state
[2019-08-25 00:37:38,252] 307 INFO MainThread cinder_helper.create_volume:: Volume is created and in available state: 6a47ae76-a2d1-4db6-bde6-44622c83809c
[2019-08-25 00:37:38,252] 957 INFO MainThread vm_helper.boot_vm :: Booting VM tenant1-virtio-0-1 with args: --boot-volume=6a47ae76-a2d1-4db6-bde6-44622c83809c --flavor=24f9af18-435c-4198-9ee0-29c15a92d4eb --poll --key-name=keypair-tenant1 --nic net-id=d3602b78-6c37-4a67-84de-7b9efdbe072b --nic port-id=e2c0bea7-e127-4d65-a055-2476a095eb25 --nic port-id=9e8d5f81-c1f9-4dbc-aa5f-bfa7859bf6ca tenant1-virtio-0-1
[2019-08-25 00:37:38,252] 1534 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for wcp_3_6
[2019-08-25 00:37:38,252] 466 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-08-25 00:37:38,252] 301 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --boot-volume=6a47ae76-a2d1-4db6-bde6-44622c83809c --flavor=24f9af18-435c-4198-9ee0-29c15a92d4eb --poll --key-name=keypair-tenant1 --nic net-id=d3602b78-6c37-4a67-84de-7b9efdbe072b --nic port-id=e2c0bea7-e127-4d65-a055-2476a095eb25 --nic port-id=9e8d5f81-c1f9-4dbc-aa5f-bfa7859bf6ca tenant1-virtio-0-1'
[2019-08-25 00:37:39,985] 423 DEBUG MainThread ssh.expect :: Output:
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.CinderConnectionFailed'> (HTTP 500) (Request-ID: req-6796326e-f489-4d40-b21c-9d224133480f)
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-08-25 00:37:39,985] 301 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-08-25 00:37:40,088] 423 DEBUG MainThread ssh.expect :: Output:
1
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-08-25 00:37:40,089] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line: ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

   3.
Expected Behavior
------------------
No exception launch of VM
Actual Behavior
----------------
As per description exception

Reproducibility
---------------
Not sure tested once seen
System Configuration
--------------------
2+2 system
Branch/Pull Time/Commit
-----------------------
BUILD_DATE= 2019-08-23_20-59-00

Last Pass
---------
2019-08-09_20-59-00 on storage system

Timestamp/Logs
--------------
[2019-08-25 00:37:38,252]
Test Activity
-------------
Regression test

[Frank Miller (sensfan22)]

As this issue is intermittent and not easily reproducible (passed in other regressions), making stx.3.0 gating and medium priority.

Assigning to distro.openstack PL to assign a prime for this bug.

[Peng Peng (ppeng)]

Issue was reproduced on
Lab: WCP_112
Load: 20190908T230000Z

[2019-09-09 18:40:59,867] 301 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --flavor=c4bd4d7a-7684-475b-96cf-f069dce85c13 --poll --boot-volume=77dcad99-59fc-48a7-ab2b-85b136c59088 --key-name=keypair-tenant1 --nic net-id=a06a050c-b3c0-474f-a1ac-6d8058a12318 tenant1-tis-centos-guest_vifs-1'
[2019-09-09 18:41:01,671] 423 DEBUG MainThread ssh.expect :: Output:
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.CinderConnectionFailed'> (HTTP 500) (Request-ID: req-a76faf4d-0d41-4dd4-a0b1-a1544c542fba)
[sysadmin@controller-0 ~(keystone_admin)]$

[Yang Liu (yliu12)]

Seen again on 20191018T013000Z load.
This is consistently reproducible on https system when booting vm using nova cmd from existing cinder volume.

[yong hu (yhu6)]

@martin, please verify this LP after OpenStack "Train" is there in StarlingX

[chen haochuan (martin1982)]

not reproduce with reproduce step in nova_boot.txt

[chen haochuan (martin1982)]

reproduce with 2019-11-20 image

[chen haochuan (martin1982)]

reproduce with openstack train. Could not produce, propose to close this issue.

[Peng Peng (ppeng)]

Issue was not reproduced on train
2019-11-21_20-00-00
wcp_3-6

[Ghada Khalil (gkhalil)]

Closing as per above; issue is not reproducible in openstack train