vm_state become ERROR after execute 'nova set-password <server-id>' failed

Bug #1757061 reported by tangxing on 2018-03-20
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
s10
Ocata
Medium
Unassigned
Pike
Medium
s10
Queens
Medium
s10
Rocky
Medium
s10

Bug Description

Description
===========
Virtual Machine's vm_state become ERROR when execute 'nova set-password <server-id>' failed.In fact,
the virtual machine is normal.

Steps to reproduce
==================
[root@nail-5300-1 ~(keystone_admin)]# nova set-password test_tx
New password:
Again:
ERROR (Conflict): Failed to set admin password on cfa4ae9f-346f-46fe-984e-08d050d3a2fc because error setting admin password (HTTP 409) (Request-ID: req-77e9f49b-0fc7-40d5-8112-9da965b0304d)
[root@nail-5300-1 ~(keystone_admin)]# nova list
+--------------------------------------+-------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+---------------------+
| cfa4ae9f-346f-46fe-984e-08d050d3a2fc | test_tx | ERROR | - | Running | HAHAHA=192.168.0.12 |
+--------------------------------------+-------------+--------+------------+-------------+---------------------+
[root@nail-5300-1 ~(keystone_admin)]#
[root@nail-5300-1 ~(keystone_admin)]#
[root@nail-5300-1 ~(keystone_admin)]#
[root@nail-5300-1 ~(keystone_admin)]# nova show test_tx|grep instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000011 |
[root@nail-5300-1 ~(keystone_admin)]# ssh nail-5300-2
Last login: Tue Mar 20 15:14:52 2018 from 10.43.203.85
[root@nail-5300-2 ~]# virsh list
 Id Name State
----------------------------------------------------
 274 instance-00000011 running

Expected result:
Set admin password failed,but virtual machine's vm_state is ACTIVE.
Actual result
=============
virtual machine's vm_state is ERROR.

tangxing (tang-xing) on 2018-03-20
Changed in nova:
milestone: none → next
Deepak Mourya (mourya007) wrote :

I am checking the bug on my Pike environment where i have got the error.

nova set-password test_vm
ubNew password:
Again:

ERROR (Conflict): QEMU guest agent is not enabled (HTTP 409) (Request-ID: req-e5fd2caf-7717-4cae-9405-5e24430e8979)

tangxing (tang-xing) wrote :

if raise except as follow: InstanceAgentNotEnabled,SetAdminPasswdNotSupported,NotImplementedError,NotImplementedError,UnexpectedTaskStateError,the vm's state is ACTIVE,
but vm's vm_state become ERROR if raise other except.

jichenjc (jichenjc) wrote :

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3522

other exception can lead to error state
what's the error in the compute log might be helpful .....

Changed in nova:
status: New → Incomplete
tangxing (tang-xing) wrote :

I think it's OK to give users a hint of information,rather than the virtual machine state is set to error.This led to the user cannot continue to operate the virtual machine, virtual machine also can work, just modify the password failed.

Changed in nova:
status: Incomplete → New
jichenjc (jichenjc) wrote :

fully understand the request, however, better to provide what's wrong there in order to decide based on more solid information..

in your case, you can use nova reset --active xxx to make instance back to normal

jichenjc (jichenjc) wrote :

did another check ...seems set_admin_password is a 'call' instead of 'cast', so looks like we can keep it as ACTIVE, let me try work on it

Changed in nova:
assignee: nobody → jichenjc (jichenjc)
status: New → Confirmed
tangxing (tang-xing) on 2018-03-22
Changed in nova:
importance: Undecided → Low

Fix proposed to branch: master
Review: https://review.openstack.org/555160

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
assignee: jichenjc (jichenjc) → s10 (vlad-esten)
Matt Riedemann (mriedem) on 2018-10-03
Changed in nova:
assignee: s10 (vlad-esten) → jichenjc (jichenjc)
tags: added: compute
Changed in nova:
assignee: jichenjc (jichenjc) → s10 (vlad-esten)
s10 (vlad-esten) on 2018-10-03
Changed in nova:
assignee: s10 (vlad-esten) → nobody
Matt Riedemann (mriedem) on 2018-10-03
Changed in nova:
importance: Low → Medium
assignee: nobody → jichenjc (jichenjc)
Changed in nova:
assignee: jichenjc (jichenjc) → s10 (vlad-esten)
Matt Riedemann (mriedem) on 2018-10-03
Changed in nova:
assignee: s10 (vlad-esten) → jichenjc (jichenjc)
Changed in nova:
assignee: jichenjc (jichenjc) → s10 (vlad-esten)

Reviewed: https://review.openstack.org/555160
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=513f2d3d254e2ffcc5c9eb786bc1c7d52036d392
Submitter: Zuul
Branch: master

commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/608165
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae39c0332d773226e3bd7dd0f81da9660b7ba7cb
Submitter: Zuul
Branch: stable/rocky

commit ae39c0332d773226e3bd7dd0f81da9660b7ba7cb
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061
    (cherry picked from commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392)

This issue was fixed in the openstack/nova 18.0.3 release.

Reviewed: https://review.openstack.org/608179
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4ac57b5aca18a91c6f29fe9c1a8fb0456c8c7785
Submitter: Zuul
Branch: stable/queens

commit 4ac57b5aca18a91c6f29fe9c1a8fb0456c8c7785
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061
    (cherry picked from commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392)

This issue was fixed in the openstack/nova 17.0.9 release.

Reviewed: https://review.openstack.org/608180
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=58ed9bc606692c140d1cb315e83795b2e73ade9d
Submitter: Zuul
Branch: stable/pike

commit 58ed9bc606692c140d1cb315e83795b2e73ade9d
Author: jichen <email address hidden>
Date: Thu Mar 22 14:07:20 2018 +0800

    Not set instance to ERROR if set_admin_password failed

    In some cases, an instance will be set to ERROR state when
    set_admin_password failed (some Exception like Forbidden)
    this is inconsistent to other exceptions and also
    set_admin_password is a sync call from API to compute,
    we can simply return the error to the upper layer (operator or
    user) to avoid make user run reset to restore instance
    status since no changes to guest at all.

    Change-Id: If1c901b974bc7295927b3f033a04eaa6ac36f603
    Closes-Bug: 1757061
    (cherry picked from commit 513f2d3d254e2ffcc5c9eb786bc1c7d52036d392)

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

This issue was fixed in the openstack/nova 16.1.8 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers