vm_state become ERROR after execute 'nova set-password <server-id>' failed

[tangxing]

Description
===========
Virtual Machine's vm_state become ERROR when execute 'nova set-password <server-id>' failed.In fact,
the virtual machine is normal.

Steps to reproduce
==================
[root@nail-5300-1 ~(keystone_admin)]# nova set-password test_tx
New password:
Again:
ERROR (Conflict): Failed to set admin password on cfa4ae9f-346f-46fe-984e-08d050d3a2fc because error setting admin password (HTTP 409) (Request-ID: req-77e9f49b-0fc7-40d5-8112-9da965b0304d)
[root@nail-5300-1 ~(keystone_admin)]# nova list
+--------------------------------------+-------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+---------------------+
| cfa4ae9f-346f-46fe-984e-08d050d3a2fc | test_tx | ERROR | - | Running | HAHAHA=192.168.0.12 |
+--------------------------------------+-------------+--------+------------+-------------+---------------------+
[root@nail-5300-1 ~(keystone_admin)]#
[root@nail-5300-1 ~(keystone_admin)]#
[root@nail-5300-1 ~(keystone_admin)]#
[root@nail-5300-1 ~(keystone_admin)]# nova show test_tx|grep instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000011 |
[root@nail-5300-1 ~(keystone_admin)]# ssh nail-5300-2
Last login: Tue Mar 20 15:14:52 2018 from 10.43.203.85
[root@nail-5300-2 ~]# virsh list
 Id Name State
----------------------------------------------------
 274 instance-00000011 running

Expected result:
Set admin password failed,but virtual machine's vm_state is ACTIVE.
Actual result
=============
virtual machine's vm_state is ERROR.

[Deepak Mourya (mourya007)]

I am checking the bug on my Pike environment where i have got the error.

nova set-password test_vm
ubNew password:
Again:

ERROR (Conflict): QEMU guest agent is not enabled (HTTP 409) (Request-ID: req-e5fd2caf-7717-4cae-9405-5e24430e8979)

[tangxing (tang-xing)]

if raise except as follow: InstanceAgentNotEnabled,SetAdminPasswdNotSupported,NotImplementedError,NotImplementedError,UnexpectedTaskStateError,the vm's state is ACTIVE,
but vm's vm_state become ERROR if raise other except.

[jichenjc (jichenjc)]

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3522

other exception can lead to error state
what's the error in the compute log might be helpful .....

[tangxing (tang-xing)]

I think it's OK to give users a hint of information,rather than the virtual machine state is set to error.This led to the user cannot continue to operate the virtual machine, virtual machine also can work, just modify the password failed.

[jichenjc (jichenjc)]

fully understand the request, however, better to provide what's wrong there in order to decide based on more solid information..

in your case, you can use nova reset --active xxx to make instance back to normal

[jichenjc (jichenjc)]

did another check ...seems set_admin_password is a 'call' instead of 'cast', so looks like we can keep it as ACTIVE, let me try work on it

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/555160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555160
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=513f2d3d254e2ffcc5c9eb786bc1c7d52036d392
Submitter: Zuul
Branch: master

commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/608165

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/608179

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/608180

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608165
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae39c0332d773226e3bd7dd0f81da9660b7ba7cb
Submitter: Zuul
Branch: stable/rocky

commit ae39c0332d773226e3bd7dd0f81da9660b7ba7cb
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061
    (cherry picked from commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.3 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608179
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4ac57b5aca18a91c6f29fe9c1a8fb0456c8c7785
Submitter: Zuul
Branch: stable/queens

commit 4ac57b5aca18a91c6f29fe9c1a8fb0456c8c7785
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061
    (cherry picked from commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.9 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608180
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=58ed9bc606692c140d1cb315e83795b2e73ade9d
Submitter: Zuul
Branch: stable/pike

commit 58ed9bc606692c140d1cb315e83795b2e73ade9d
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061
    (cherry picked from commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.8 release.