Reaching VM limit creation (10) virsh doesn't list all running instances

[Ricardo Perez]

Brief Description
-----------------

Running a duplex StarlingX system configuration, if reaching the VM limit creation (10 instances) using CirrOS / m1.tiny flavor the system will create them, however, besides you can see all of them in an "ACTIVE" state, just few of them will be listed by virsh command, and only the ones listed by virsh will be accesible via console session.

Severity
--------
<Major: System/Feature is usable but degraded>

Steps to Reproduce
------------------
1.- Follow the instructions described here to set up a StarlingX duplex configuration:
https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnAIODX

2.- Download a CirrOS image from here:
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

3.- Create an Image (as root user _not_ as keystone_admin)
openstack image create --container-format bare --disk-format qcow2 --file cirros-0.4.0-x86_64-disk.img cirros

4.- Create a VM
openstack server create --image cirros --flavor m1.tiny --network public-net0 vm1

5.- Continue creating VMs, until you reach the limit, you will see a message like:

controller-0:~# openstack server create --image cirros --flavor m1.tiny --network public-net0 richovm10
Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) (Request-ID: req-883f68d0-6b78-4aba-b1ab-dc2f53b54069)

6.- List all created VMs with "openstack server list"

controller-0:~# openstack server list
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ec9ff19a-3dc6-409a-aadb-d260632355e4 | richovm9 | ACTIVE | public-net0=192.168.101.228 | cirros | m1.tiny |
| 8ecdb032-e0da-40ee-b84b-57f314a1e3f3 | richovm8 | ACTIVE | public-net0=192.168.101.71 | richo | m1.richo |
| e3000c0f-4211-4b89-b3e6-362a04ba56db | richovm7 | ACTIVE | public-net0=192.168.101.140 | richo | m1.richo |
| 8a38cce5-deee-4b77-b18d-f86a7653dd47 | richovm6 | ACTIVE | public-net0=192.168.101.86 | richo | m1.richo |
| d7d378d9-26c0-438c-a4a3-95fa85126041 | richovm5 | ACTIVE | public-net0=192.168.101.15 | richo | m1.richo |
| 43ddac80-baf2-49d3-982a-a23228a5eebc | richovm4 | ACTIVE | public-net0=192.168.101.118 | richo | m1.richo |
| 3073812e-6ebc-44ff-bddf-a46c4b499c11 | richovm3 | ACTIVE | public-net0=192.168.101.195 | richo | m1.richo |
| 60097df8-2b3b-4540-85f5-25808a27c43f | richovm2 | ACTIVE | public-net0=192.168.101.116 | richo | m1.richo |
| 8d26efc3-454a-44b1-9222-bf5eecf711a0 | richovm | ACTIVE | public-net0=192.168.101.134 | richo | m1.richo |
| cc1e49f6-b096-41db-964d-58b86eadcc3e | vm1 | ACTIVE | public-net0=192.168.101.218 | cirros | m1.tiny |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+

7.- List all available consoles using virsh command:

controller-0:~# sudo virsh list --all
 Id Name State
-----------------------------------
 3 instance-00000004 running
 4 instance-00000008 running
 5 instance-0000000c running
 6 instance-00000010 running
 7 instance-00000014 running

8.- Access a CirrOS console using "virsh console". You can do it only for virsh listed VMs.
virsh console <VM_ID_number>

9.- Try to login to the remain (not listed by virsh command) VM's using this alternative step. You can use "openstack server show" to see the instance name (used by virsh) for each one of the running VMS, then use "virsh console <Instance_VM_Nam> just to see wich instance-XXXXX number correspond to a openstack VM name.

controller-0:~# openstack server list
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ec9ff19a-3dc6-409a-aadb-d260632355e4 | richovm9 | ACTIVE | public-net0=192.168.101.228 | cirros | m1.tiny |

controller-0:~# openstack server show ec9ff19a-3dc6-409a-aadb-d260632355e4
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | controller-1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-1 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000018 |
| OS-EXT-STS:power_state | Running |

controller-0:~# sudo virsh console instance-00000018
error: failed to get domain 'instance-00000018'
error: Domain not found: no domain with matching name 'instance-00000018'

Expected Behavior
------------------

virsh list should list all running VM instances.

Actual Behavior
----------------

virsh list list only few of the instances running, only 5 of 10 created VMs.

Reproducibility
---------------
<Reproducible> 100% of the times.

System Configuration
--------------------
<Two node system> Duplex

Branch/Pull Time/Commit
-----------------------
controller-0:~# cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190630T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="166"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-30 23:30:00 +0000"
controller-0:~#

Last Pass
---------
Not known.

Timestamp/Logs
--------------

controller-0:~# openstack server list
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ec9ff19a-3dc6-409a-aadb-d260632355e4 | richovm9 | ACTIVE | public-net0=192.168.101.228 | cirros | m1.tiny |
| 8ecdb032-e0da-40ee-b84b-57f314a1e3f3 | richovm8 | ACTIVE | public-net0=192.168.101.71 | richo | m1.richo |
| e3000c0f-4211-4b89-b3e6-362a04ba56db | richovm7 | ACTIVE | public-net0=192.168.101.140 | richo | m1.richo |
| 8a38cce5-deee-4b77-b18d-f86a7653dd47 | richovm6 | ACTIVE | public-net0=192.168.101.86 | richo | m1.richo |
| d7d378d9-26c0-438c-a4a3-95fa85126041 | richovm5 | ACTIVE | public-net0=192.168.101.15 | richo | m1.richo |
| 43ddac80-baf2-49d3-982a-a23228a5eebc | richovm4 | ACTIVE | public-net0=192.168.101.118 | richo | m1.richo |
| 3073812e-6ebc-44ff-bddf-a46c4b499c11 | richovm3 | ACTIVE | public-net0=192.168.101.195 | richo | m1.richo |
| 60097df8-2b3b-4540-85f5-25808a27c43f | richovm2 | ACTIVE | public-net0=192.168.101.116 | richo | m1.richo |
| 8d26efc3-454a-44b1-9222-bf5eecf711a0 | richovm | ACTIVE | public-net0=192.168.101.134 | richo | m1.richo |
| cc1e49f6-b096-41db-964d-58b86eadcc3e | vm1 | ACTIVE | public-net0=192.168.101.218 | cirros | m1.tiny |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
controller-0:~# openstack server create --image cirros --flavor m1.tiny --network public-net0 richovm10
Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) (Request-ID: req-883f68d0-6b78-4aba-b1ab-dc2f53b54069)

controller-0:~# sudo virsh list
 Id Name State
-----------------------------------
 3 instance-00000004 running
 4 instance-00000008 running
 5 instance-0000000c running
 6 instance-00000010 running
 7 instance-00000014 running

Test Activity
-------------
[Regression Testing]

[Ricardo Perez (richomx)]

All nodes full log.

[Ricardo Perez (richomx)]

Only VMs living in the compute-0 are the ones listed by virsh, the VMs living in the controller-1 aren't listed. Please check the following log:

controller-0:~# openstack server list
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
| ec9ff19a-3dc6-409a-aadb-d260632355e4 | richovm9 | ACTIVE | public-net0=192.168.101.228 | cirros | m1.tiny |
| 8ecdb032-e0da-40ee-b84b-57f314a1e3f3 | richovm8 | ACTIVE | public-net0=192.168.101.71 | richo | m1.richo |
| e3000c0f-4211-4b89-b3e6-362a04ba56db | richovm7 | ACTIVE | public-net0=192.168.101.140 | richo | m1.richo |
| 8a38cce5-deee-4b77-b18d-f86a7653dd47 | richovm6 | ACTIVE | public-net0=192.168.101.86 | richo | m1.richo |
| d7d378d9-26c0-438c-a4a3-95fa85126041 | richovm5 | ACTIVE | public-net0=192.168.101.15 | richo | m1.richo |
| 43ddac80-baf2-49d3-982a-a23228a5eebc | richovm4 | ACTIVE | public-net0=192.168.101.118 | richo | m1.richo |
| 3073812e-6ebc-44ff-bddf-a46c4b499c11 | richovm3 | ACTIVE | public-net0=192.168.101.195 | richo | m1.richo |
| 60097df8-2b3b-4540-85f5-25808a27c43f | richovm2 | ACTIVE | public-net0=192.168.101.116 | richo | m1.richo |
| 8d26efc3-454a-44b1-9222-bf5eecf711a0 | richovm | ACTIVE | public-net0=192.168.101.134 | richo | m1.richo |
| cc1e49f6-b096-41db-964d-58b86eadcc3e | vm1 | ACTIVE | public-net0=192.168.101.218 | cirros | m1.tiny |
+--------------------------------------+----------+--------+-----------------------------+--------+----------+
controller-0:~# openstack server show vm1 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
controller-0:~# openstack server show richovm | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-1 |
controller-0:~# openstack server show richovm2 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
controller-0:~# openstack server show richovm3 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-1 |
controller-0:~# openstack server show richovm4 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
controller-0:~# openstack server show richovm5 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-1 |
controller-0:~# openstack server show richovm6 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
controller-0:~# openstack server show richovm7 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-1 |
controller-0:~# openstack server show richovm8 | grep hypervisor
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 ...

[Ricardo Perez (richomx)]

Doing verification of the problem reported, I saw that the behavior of virsh command is as follows:

It only lists the living VMs on the logged controller (in this case is a Duplex configuration).

* For controller 1, it only shows 5 living VMs using ' virsh list '.
* For controller 2, it shows the other 5 living VMs using the same command.

Is this the expected behavior, even if using "virsh list --all" it will only show the VMs that has been created in the corresponding controller.

Doing a manual migration of the instances living in the controler-1 they can be listed in controller-0 using "virsh list". So it seems like this is the normal behavior of the command.

Unless someone needs more information about this, this bug can be closed as invalid.

[Ghada Khalil (gkhalil)]

Marking as Low / not gating based on input from the reporter that this is not a valid issue