nova hypervisor-stats displays 0 instead of expected values

[Jatin Mehta]

1. Cloned latest vanilla devstack with latest Mitaka code and used pxe_ssh driver to create ironic-node.

ironic node-create --name NODE -d pxe_ssh

2. Updated node with "required" pxe_ssh attributes

ironic node-update $NODE_UUID add \
driver_info/deploy_kernel=$DEPLOY_VMLINUZ_UUID \
driver_info/deploy_ramdisk=$DEPLOY_INITRD_UUID \
driver_info/ssh_virt_type="virsh"

ironic node-update $NODE_UUID add instance_info/image_source=$IMG instance_info/root_gb=10
ironic node-update $NODE_UUID add \
driver_info/ssh_username=stcsdn_admin \
driver_info/ssh_password=quantumquantum \
driver_info/ssh_address=105.0.2.149

3. nova hypervisor-list shows the node created

4. nova hypervisor-stats lists everything as 0 despite specifications for flavor creation

as follows:
RAM_MB=1028
CPU=2
DISK_GB=100
ARCH='{i686|x86_64}'

Updated node with above properties:
ironic node-update $NODE_UUID add \
properties/cpus=$CPU \
properties/memory_mb=$RAM_MB \
properties/local_gb=$DISK_GB \
properties/cpu_arch=$ARCH

 nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 0 |
| free_disk_gb | 0 |
| free_ram_mb | 0 |
| local_gb | 0 |
| local_gb_used | 0 |
| memory_mb | 0 |
| memory_mb_used | 0 |
| running_vms | 0 |
| vcpus | 0 |
| vcpus_used | 0 |
+----------------------+-------+

Anyone knows the reason why is it listing so? I tried using pxe_ipmitool and created ironic node via it, the hypervisor-stats came up just fine.

nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 100 |
| free_disk_gb | 100 |
| free_ram_mb | 1028 |
| local_gb | 100 |
| local_gb_used | 0 |
| memory_mb | 1028 |
| memory_mb_used | 0 |
| running_vms | 0 |
| vcpus | 2 |
| vcpus_used | 0 |
+----------------------+-------+

NOTE: The behavior was also seen for fake_agent and fake drivers.
As a result, In spite of ironic node creation having same resource configured as the flavor used while booting the BM fails since nova scheduler filters the hypervisor with the flavor of BM and says that no such hypervisor found. Bm goes to error state.

[Haomeng,Wang (whaom)]

Thanks Jatin to raise this issue, I will try to recreate and investigate this issue, and update the result here.

[Jim Rollenhagen (jim-rollenhagen)]

Can you paste the output of "ironic node-show $uuid" for that node, please?

The ironic driver doesn't report resources for a node in certain states, like maintenance mode, any provision state != AVAILABLE, etc. See also https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L330-L351

I know pxe_ssh works since that's what we use for gate testing :)

[Jatin Mehta (jmehta)]

Here you go, Jim!

ironic node-show jatin
+------------------------+-----------------------------------------------------------------------+
| Property | Value |
+------------------------+-----------------------------------------------------------------------+
| target_power_state | None |
| extra | {} |
| last_error | None |
| updated_at | 2015-11-05T18:18:11+00:00 |
| maintenance_reason | None |
| provision_state | available |
| clean_step | {} |
| uuid | eec902e1-16c6-4f04-9990-d1ef75149710 |
| console_enabled | False |
| target_provision_state | None |
| provision_updated_at | None |
| maintenance | False |
| inspection_started_at | None |
| inspection_finished_at | None |
| power_state | None |
| driver | pxe_ssh |
| reservation | None |
| properties | {u'memory_mb': 1028, u'cpu_arch': u'{i686|x86_64}', u'local_gb': 100, |
| | u'cpus': 2, u'capabilities': u'boot_option:local'} |
| instance_uuid | None |
| name | jatin |
| driver_info | {u'ssh_username': u'stcsdn_admin', u'deploy_kernel': u'3984971d- |
| | f17d-4481-af1a-975498a3459e', u'deploy_ramdisk': u'2e2d9bfb- |
| | 21a3-4082-aba5-e97fec11d903', u'ssh_password': u'******', |
| | u'ssh_virt_type': u'virsh', u'ssh_address': u'105.0.2.149'} |
| created_at | 2015-11-05T18:18:06+00:00 |
| driver_internal_info | {} |
| chassis_uuid | |
| instance_info | {u'root_gb': 10, u'image_source': ...

[Haomeng,Wang (whaom)]

Hi Jatin,

Try to recreate this issue two times, can not reproduced, I just run devstack to deploy openstack env, so what is your steps?

ubuntu@ubuntu-devstack:~/devstack$ nova hypervisor-show 594b2228-9a80-4902-936d-10cdad35d21f
+-------------------------+--------------------------------------+
| Property | Value |
+-------------------------+--------------------------------------+
| cpu_info | |
| current_workload | 0 |
| disk_available_least | 10 |
| free_disk_gb | 10 |
| free_ram_mb | 1024 |
| host_ip | 9.5.126.32 |
| hypervisor_hostname | 594b2228-9a80-4902-936d-10cdad35d21f |
| hypervisor_type | ironic |
| hypervisor_version | 1 |
| id | 1 |
| local_gb | 10 |
| local_gb_used | 0 |
| memory_mb | 1024 |
| memory_mb_used | 0 |
| running_vms | 0 |
| service_disabled_reason | None |
| service_host | ubuntu-devstack |
| service_id | 5 |
| state | up |
| status | enabled |
| vcpus | 1 |
| vcpus_used | 0 |
+-------------------------+--------------------------------------+
ubuntu@ubuntu-devstack:~/devstack$ ironic node-show 594b2228-9a80-4902-936d-10cdad35d21f
+------------------------+-------------------------------------------------------------------------+
| Property | Value |
+------------------------+-------------------------------------------------------------------------+
| target_power_state | None |
| extra | {} |
| last_error | None |
| updated_at | 2015-11-09T08:18:36+00:00 |
| maintenance_reason | None |
| provision_state | available |
| clean_step | {} |
| uuid | 594b2228-9a80-4902-936d-10cdad35d21f |
| console_enabled | ...

[Dmitry Tantsur (divius)]

> u'cpu_arch': u'{i686|x86_64}',

this looks incorrect, please try with this value fixed

[Jatin Mehta (jmehta)]

I tried with x86_64 too (I think that is what you mean by correct value),

| properties | {u'memory_mb': 1024, u'cpu_arch': u'x86_64', u'local_gb': 10, u'cpus': |
| | 1}

It makes no difference to the issue. The steps I take are

1. clone a fresh setup for devstack with ironic enabled as per http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack
2. The ironic nodes that are created via the devstack script according to the details mentioned in local.conf :
IRONIC_VM_COUNT=2
IRONIC_VM_SSH_PORT=22
IRONIC_BAREMETAL_BASIC_OPS=True
These ironic nodes are created via pxe_ssh driver. Their hypervisor stats display fine. But when I used the same command from devstack.log to create a new ironic node, it's hypervisor stats showed stats as 0.
Here is the command I use from devstack.log

ironic node-create --uuid 902abe6b-b2d4-4363-ae05-21ffe2cf9923 --chassis_uuid f648dc73-f07a-423c-a203-869f1c23b5fd --driver pxe_ssh --name node-0 -p cpus=1 -p memory_mb=1024 -p local_gb=10 -p cpu_arch=x86_64 -i deploy_kernel=0c7d4fd8-e0e9-488a-992b-51d65a65a090 -i deploy_ramdisk=fb107aa9-56e5-4a3c-b4a1-c101c9622652 -i ssh_virt_type=virsh -i ssh_address=105.0.1.78 -i ssh_port=22 -i ssh_username=stack -i ssh_key_filename=/opt/stack/data/ironic/ssh_keys/ironic_key

[Jatin Mehta (jmehta)]

Hi guys,

I figured out the problem. If an ironic node is not associated with an ironic port with a VALID MAC ADDRESS, it will show hypervisor stats zero. Not sure if this is a bug. As per ir-cond.log,
 SSH driver was not able to find a VM with any of the specified MACs: [u'52:54:00:9d:85:9f'] for node 2235a4ed-0a64-4f6a-95d2-2e74389708d9..

Now my main purpose of using pxe_ssh was to work with fake bare metals. Thus, I was providing with a fake MAC ID which was taken as invalid and thus hypervisor stats stayed zero.
So, I tried using the MAC ID of the baremetalbrbm_0 which is a fake BM created during stack.sh by copying its MAC from the xml file of the BM.

virsh dumpxml baremetalbrbm_0 | grep mac
    <type arch='x86_64' machine='pc-1.0'>hvm</type>
      <mac address='52:54:00:13:fe:be'/>
If I use this MAC address to create an ironic port, the associated ironic node displays proper hypervisor stats. No Problems! I use this command:
ironic port-create -a 52:54:00:13:fe:be -n 2a6bc8fa-28e3-467d-a10b-f6d4c48d3ec4
 nova hypervisor-show 2a6bc8fa-28e3-467d-a10b-f6d4c48d3ec4
+-------------------------+--------------------------------------+
| Property | Value |
+-------------------------+--------------------------------------+
| cpu_info | |
| current_workload | 0 |
| disk_available_least | 10 |
| free_disk_gb | 10 |
| free_ram_mb | 1024 |
| host_ip | 105.0.1.78 |
| hypervisor_hostname | 2a6bc8fa-28e3-467d-a10b-f6d4c48d3ec4 |
| hypervisor_type | ironic |
| hypervisor_version | 1 |
| id | 11 |
| local_gb | 10 |
| local_gb_used | 0 |
| memory_mb | 1024 |
| memory_mb_used | 0 |
| running_vms | 0 |
| service_disabled_reason | None |
| service_host | ironicon78 |
| service_id | 6 |
| state | up |
| status | enabled |
| vcpus | 1 |
| vcpus_used | 0 |
+-------------------------+--------------------------------------+

When I create a new baremetal in the same way that stack.sh did, it is created successfully. Used the following command:
 sudo su stack -c '/home/stack/devstack/tools/ironic/scripts/create-node baremetalbrbm_0123 1 1024 10 amd64 brbm /usr/bin/qemu-system-x86_64 /opt/stack/ironic-bm-logs'

But when I try to use the MAC of this newly...

[Jatin Mehta (jmehta)]

Hi guys!

I finally resolved the issue. It is certainly not a bug. All I had to do for my simulated bare metal was to start them via virsh. It seems that the MAC was invalid as long as you don't have the simulated BM as started. As soon as I did a

virsh start <domain_id_for_simulated_baremetal>

the nova hypervisor-show <ironic_node_uuid> started showing proper hypervisor-stats after a couple of minutes.
Please change the status to appropriate one as this thread might be helpful to someone else performing similar tasks.

[Dmitry Tantsur (divius)]

Thank you for update.