server show failed by "No server with a name" after VM boot up

[Peng Peng]

Brief Description
-----------------
After VM boots up successfully, server show failed by "No server with a name or ID"

Severity
--------
Major

Steps to Reproduce
------------------
as descriiption

TC-name: test_multiple_ports.py::TestMutiPortsBasic::test_multiports_on_same_network_vm_actions[virtio_x4]

Expected Behavior
------------------
server show cmd work

Actual Behavior
----------------

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
One node system

Lab-name: SM_4

Branch/Pull Time/Commit
-----------------------
20191210T000000Z

Last Pass
---------
2019-11-25_20-00-00

Timestamp/Logs
--------------
[2019-12-10 10:00:54,598] 311 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --poll --boot-volume=5b69c291-2f56-4220-a720-015afd9cad4a --flavor=7f07adeb-9b6b-417b-bd6b-202ee7f738cf --key-name=keypair-tenant1 --nic net-id=6c1258cd-93bf-4a82-a64e-4bc6e833da79 --nic port-id=6969d691-003e-4e7d-91b8-c4e936bc102f --nic port-id=02e8a6ca-1192-4615-a792-c915272dc230 --nic port-id=b2a9ffdd-302a-4dc9-9780-eee09f7960c8 --nic port-id=55fb2e1d-83e8-432e-a4d6-a4326cad4864 tenant1-multiports-2'
[2019-12-10 10:02:06,696] 433 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+-------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | ek9NsEfbHMXR |
| config_drive | |
| created | 2019-12-10T10:01:06Z |
| description | - |
| flavor:disk | 2 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "large"} |
| flavor:original_name | dedicated |
| flavor:ram | 1024 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| id | 01db7a12-a7b5-445c-a6fc-47afe518d9e5 |
| image | Attempt to boot from volume - no image supplied |
| key_name | keypair-tenant1 |
| locked | False |
| metadata | {} |
| name | tenant1-multiports-2 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 6c114d77663a422fad814d314528f09e |
| trusted_image_certificates | - |
| updated | 2019-12-10T10:01:06Z |
| user_id | 8afe5b1a4bd543d2b18ae2cf2fda8504 |
+--------------------------------------+-------------------------------------------------------------+

Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 100% complete
Finished
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-12-10 10:02:06,697] 311 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-12-10 10:02:06,803] 433 DEBUG MainThread ssh.expect :: Output:
0
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-12-10 10:02:06,803] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line:
[2019-12-10 10:02:06,803] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line:
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 100% complete
[2019-12-10 10:02:06,804] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line: Finished
[2019-12-10 10:02:06,804] 983 INFO MainThread vm_helper.boot_vm :: Post action check...
[2019-12-10 10:02:06,804] 1599 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for sm_4
[2019-12-10 10:02:06,804] 476 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-12-10 10:02:06,804] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show 01db7a12-a7b5-445c-a6fc-47afe518d9e5'
[2019-12-10 10:04:20,329] 433 DEBUG MainThread ssh.expect :: Output:
No server with a name or ID of '01db7a12-a7b5-445c-a6fc-47afe518d9e5' exists.
[sysadmin@controller-0 ~(keystone_admin)]$

[2019-12-10 10:23:43,918] 311 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --image=ef39f9ae-04f9-45ca-8e3a-dc48ac888e4d --availability-zone=nova --poll --flavor=c8c79ccd-c0d5-4456-988b-b937a9950cc7 --key-name=keypair-tenant1 --nic net-id=6c1258cd-93bf-4a82-a64e-4bc6e833da79 --nic net-id=e2526161-29d1-461b-8e68-6e6ddd853ce7 tenant1-image_root_attachvol-7'
[2019-12-10 10:25:04,239] 433 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+---------------------------------------------------------+
| Property | Value |
+--------------------------------------+---------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | zhbN34vgGjve |
| config_drive | |
| created | 2019-12-10T10:23:47Z |
| description | - |
| flavor:disk | 2 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:mem_page_size": "large"} |
| flavor:original_name | flv_rootdisk |
| flavor:ram | 1024 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| id | c8891cdc-5b06-484b-b60c-2d41ea75c47a |
| image | tis-centos-guest (ef39f9ae-04f9-45ca-8e3a-dc48ac888e4d) |
| key_name | keypair-tenant1 |
| locked | False |
| metadata | {} |
| name | tenant1-image_root_attachvol-7 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 6c114d77663a422fad814d314528f09e |
| trusted_image_certificates | - |
| updated | 2019-12-10T10:23:47Z |
| user_id | 8afe5b1a4bd543d2b18ae2cf2fda8504 |
+--------------------------------------+---------------------------------------------------------+

Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 100% complete
Finished
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-12-10 10:25:04,240] 311 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-12-10 10:25:04,343] 433 DEBUG MainThread ssh.expect :: Output:
0
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-12-10 10:25:04,344] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line:
[2019-12-10 10:25:04,344] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line:
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 100% complete
[2019-12-10 10:25:04,344] 128 DEBUG MainThread table_parser.__table :: skipping invalid table line: Finished
[2019-12-10 10:25:04,344] 983 INFO MainThread vm_helper.boot_vm :: Post action check...
[2019-12-10 10:25:04,345] 1599 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for sm_4
[2019-12-10 10:25:04,345] 476 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-12-10 10:25:04,345] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show c8891cdc-5b06-484b-b60c-2d41ea75c47a'
[2019-12-10 10:25:08,350] 433 DEBUG MainThread ssh.expect :: Output:
+-----------------------------+-----------------------------------------------------------+
| Field | Value |
+-----------------------------+-----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-12-10T10:25:00.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | tenant1-mgmt-net=192.168.96.36; tenant1-net2=172.16.2.169 |
| config_drive | |
| created | 2019-12-10T10:23:47Z |
| flavor | flv_rootdisk (c8c79ccd-c0d5-4456-988b-b937a9950cc7) |
| hostId | feccb65f427e13d57b61aef08dd0f016fd1e43ff586bec3633131ea0 |
| id | c8891cdc-5b06-484b-b60c-2d41ea75c47a |
| image | tis-centos-guest (ef39f9ae-04f9-45ca-8e3a-dc48ac888e4d) |
| key_name | keypair-tenant1 |
| name | tenant1-image_root_attachvol-7 |
| progress | 0 |
| project_id | 6c114d77663a422fad814d314528f09e |
| properties | |
| security_groups | name='default' |
| | name='default' |
| status | ACTIVE |
| updated | 2019-12-10T10:25:01Z |
| user_id | 8afe5b1a4bd543d2b18ae2cf2fda8504 |
| volumes_attached | |
+-----------------------------+-----------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$

[2019-12-10 10:25:36,877] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server add volume c8891cdc-5b06-484b-b60c-2d41ea75c47a fe5945a0-4291-40fe-834c-2b594973d552'
[2019-12-10 10:27:48,894] 433 DEBUG MainThread ssh.expect :: Output:
No server with a name or ID of 'c8891cdc-5b06-484b-b60c-2d41ea75c47a' exists.

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

Assigning to distro.openstack PL for triage.
Given that this is seen only once, I recommend it is NOT gating for stx.3.0

[yong hu (yhu6)]

suggest to move it to 4.x.

[zhipeng liu (zhipengs)]

Hi pengpeng,

Could you see this issue again during latest test?

Thanks!
Zhipeng

[Ran An (an.ran)]

Hi Pengpeng,

Since the sanity test has been green for at least a week, do you think this issue is still there? Or could you provide more info about how to reproduce this issue?

Thanks

[Ghada Khalil (gkhalil)]

Confirmed with Yang, test PL, that this has not been seen in recent testing. Closing as the issue is no longer reproducible.

[Peng Peng (ppeng)]

not seen this issue recently