When using the Ironic Nova driver, a stopped server is still presented as Running even when the server is stopped. Checking via the Ironic API correctly shows the instance as powered down:
stack@ironic:~/logs/screen$ nova list
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| 5b43d631-91e1-4384-9b87-93283b3ae958 | testing | ACTIVE | - | Running | private=10.1.0.10 |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
stack@ironic:~/logs/screen$ nova stop 5b43d631-91e1-4384-9b87-93283b3ae958
stack@ironic:~/logs/screen$ nova list
+--------------------------------------+---------+---------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+---------+------------+-------------+-------------------+
| 5b43d631-91e1-4384-9b87-93283b3ae958 | testing | SHUTOFF | - | Running | private=10.1.0.10 |
+--------------------------------------+---------+---------+------------+-------------+-------------------+
stack@ironic:~/logs/screen$ ping 10.1.0.10
PING 10.1.0.10 (10.1.0.10) 56(84) bytes of data.
From 172.24.4.2 icmp_seq=1 Destination Host Unreachable
From 172.24.4.2 icmp_seq=5 Destination Host Unreachable
From 172.24.4.2 icmp_seq=6 Destination Host Unreachable
From 172.24.4.2 icmp_seq=7 Destination Host Unreachable
From 172.24.4.2 icmp_seq=8 Destination Host Unreachable
--- 10.1.0.10 ping statistics ---
9 packets transmitted, 0 received, +5 errors, 100% packet loss, time 8000ms
stack@ironic:~/logs/screen$ ironic node-list
+--------------------------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------------------------+-------------+--------------------+-------------+
| 91e81c38-4dce-412b-8a1b-a914d28943e4 | 5b43d631-91e1-4384-9b87-93283b3ae958 | power off | active | False |
+--------------------------------------+--------------------------------------+-------------+--------------------+-------------+
I was able to reproduce this issue and did some debugging.
This issue is occurring because when the ironic driver makes an 'ironic client' call to do power action on the particular node, ironic spawns an thread to perform the power actions and the ironic client returns immediately.
Since ironic client returned, nova will query and see that the node is still in powered on state and sets the POWER STATE to running.
(we need to actually wait for the state transition to happen)
NOTE:
The POWER STATE is automatically corrected in the next 'sync power state' action that happens periodically.
But, we need to fix this issue, since user may not wait till this periodic task to happen.