nova start failed to start VM after nova stop was issued

[Peng Peng]

Brief Description
-----------------
Booting up a VM, then using "nova stop" to stop it. After VM is SHUTOFF status, using "nova start" to start the VM, but failed by action-rejected.

Severity
--------
Major

Steps to Reproduce
------------------
Booting up a VM
stop it
start it

TC-name: nova/test_nova_actions.py::test_nova_actions[ubuntu_14-shared-stop-start]

Expected Behavior
------------------
VM start successfully

Actual Behavior
----------------
VM start action rejected

Reproducibility
---------------
Seen once

System Configuration
--------------------
Multi-node system

Lab-name: IP_33-36

Branch/Pull Time/Commit
-----------------------
stx master as of 20190920T013000Z

Last Pass
---------
2019-08-22_00-10-00

Timestamp/Logs
--------------

[2019-09-20 10:18:02,007] 301 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --key-name=keypair-tenant1 --poll --boot-volume=210b7e54-1495-483e-937c-1a677f7864e4 --flavor=f14deb46-6684-45dd-9ea2-7d1d0931de7a --nic net-id=15324c7b-3571-493a-94b1-468d06ce7de2 --nic net-id=90174478-aa91-4d5f-8657-cadc2546359c tenant1-nova_actions-12'

[2019-09-20 10:18:22,966] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2019-09-20 10:18:25,272] 423 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+-------------------------+--------+------------------------------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------------+--------+------------------------------------------------------------+-------+--------+
| c3e72b06-e788-4dc7-a461-37267909bc44 | tenant1-nova_actions-12 | ACTIVE | tenant1-mgmt-net=192.168.116.36; tenant1-net0=172.16.0.187 | | shared |
+--------------------------------------+-------------------------+--------+------------------------------------------------------------+-------+--------+

[2019-09-20 10:18:32,588] 301 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne stop c3e72b06-e788-4dc7-a461-37267909bc44'
[2019-09-20 10:18:34,524] 423 DEBUG MainThread ssh.expect :: Output:
Request to stop server c3e72b06-e788-4dc7-a461-37267909bc44 has been accepted.

[2019-09-20 10:18:37,786] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2019-09-20 10:18:39,852] 423 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+-------------------------+---------+------------------------------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------------+---------+------------------------------------------------------------+-------+--------+
| c3e72b06-e788-4dc7-a461-37267909bc44 | tenant1-nova_actions-12 | SHUTOFF | tenant1-mgmt-net=192.168.116.36; tenant1-net0=172.16.0.187 | | shared |
+--------------------------------------+-------------------------+---------+------------------------------------------------------------+-------+--------+

[2019-09-20 10:18:39,957] 301 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne start c3e72b06-e788-4dc7-a461-37267909bc44'
[2019-09-20 10:18:41,489] 423 DEBUG MainThread ssh.expect :: Output:
action-rejected (HTTP 409)
ERROR (CommandError): Unable to start the specified server(s).
controller-1:~$

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

Assigning to the distro openstack PL to route to a developer for investigation

[Ghada Khalil (gkhalil)]

Marking as stx.3.0 for now until further investigation by the distro openstack team

[yong hu (yhu6)]

@Ran will investigate this LP if it's still there with OpenStack train.

[Ran An (an.ran)]

this issue could not be reproduced on iso built at 2019-11-07

controller-1:~$ openstack server list
+--------------------------------------+------+--------+----------------------------+-------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+----------------------------+-------+---------+
| 057f4c8e-e9ad-443a-94b1-14b3f98a90f2 | test | ACTIVE | public-net0=192.168.101.21 | | m1.tiny |
+--------------------------------------+------+--------+----------------------------+-------+---------+
controller-1:~$ nova --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name default --os-project-domain-name default --os-endpoint-type internalURL --os-region-name RegionOne stop 057f4c8e-e9ad-443a-94b1-14b3f98a90f2
Request to stop server 057f4c8e-e9ad-443a-94b1-14b3f98a90f2 has been accepted.
controller-1:~$
controller-1:~$ openstack server list
+--------------------------------------+------+---------+----------------------------+-------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+---------+----------------------------+-------+---------+
| 057f4c8e-e9ad-443a-94b1-14b3f98a90f2 | test | SHUTOFF | public-net0=192.168.101.21 | | m1.tiny |
+--------------------------------------+------+---------+----------------------------+-------+---------+
controller-1:~$
controller-1:~$
controller-1:~$
controller-1:~$
controller-1:~$ nova --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name default --os-project-domain-name default --os-endpoint-type internalURL --os-region-name RegionOne start 057f4c8e-e9ad-443a-94b1-14b3f98a90f2
Request to start server 057f4c8e-e9ad-443a-94b1-14b3f98a90f2 has been accepted.
controller-1:~$
controller-1:~$
controller-1:~$ openstack server list
+--------------------------------------+------+--------+----------------------------+-------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+----------------------------+-------+---------+
| 057f4c8e-e9ad-443a-94b1-14b3f98a90f2 | test | ACTIVE | public-net0=192.168.101.21 | | m1.tiny |
+--------------------------------------+------+--------+----------------------------+-------+---------+
controller-1:~$ cat /etc/build.info
SW_VERSION="19.09"
BUILD_TARGET="Unknown"
BUILD_TYPE="Informal"
BUILD_ID="n/a"

JOB="n/a"
BUILD_BY="builder"
BUILD_NUMBER="n/a"
BUILD_HOST="jenkins-starlingx-stx-eng-build-301-07rnf-r5wdf"
BUILD_DATE="2019-11-07 02:20:09 +0000"

BUILD_DIR="/"
WRS_SRC_DIR="/localdisk/designer/builder/starlingx/cgcs-root"
WRS_GIT_BRANCH="HEAD"
CGCS_SRC_DIR="/localdisk/designer/builder/starlingx/cgcs-root/stx"
CGCS_GIT_BRANCH="HEAD"

[Ran An (an.ran)]

this issue could not be reproduced on OpenStack train either.

controller-0:~$ cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.09"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20191114T195617Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="319"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-11-14 19:56:17 +0000"

[Ricardo Perez (richomx)]

This issue is no longer reproducible on Train build:

controller-0:~# cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.09"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20191122T023000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="327"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-11-22 02:30:00 +0000"
controller-0:~#

[Peng Peng (ppeng)]

[2019-11-23 01:17:04,978] 264 INFO MainThread conftest.testcase_log::
-----------------------------------------------------------------
Test Result for: testcases/functional/nova/test_nova_actions.py::test_nova_actions[ubuntu_14-shared-stop-start] - Test Passed

Issue was not reproduced on train
2019-11-21_20-00-00
wcp_3-6