EMC-POC: fail to launch VM at Compute node after the Controller+Network node reboots; intermittent failure

[Danny Choi]

In EMC-POC setup, Controller and Network node run on the same bare-metal machine.

After the Controller+Network node reboots, sometimes it fails to launch new VM at the Compute node;
though it shows the Compute node service is on-line.

To recover, reboot nova service at the Compute node or reboot Compute node.

[root@bxb-ds-46 ~]# nova-manage service list
Binary Host Zone Status State Updated_At
nova-consoleauth bxb-ds-46 internal enabled :-) 2015-06-19 14:50:23
nova-scheduler bxb-ds-46 internal enabled :-) 2015-06-19 14:50:23
nova-conductor bxb-ds-46 internal enabled :-) 2015-06-19 14:50:23
nova-cert bxb-ds-46 internal enabled :-) 2015-06-19 14:50:23
nova-compute bxb-ds-48 zone2 enabled :-) 2015-06-19 14:50:28 <<<<< on-line
nova-compute bxb-ds-47 zone1 enabled :-) 2015-06-19 14:50:23 <<<<< on-line

[root@bxb-ds-46 ~]# nova boot --image cirros-0.3.3 --flavor 1 --nic net-id=3d0ef9a2-ff8a-414a-8d56-390e03d88349 --availability-zone zone1 ds47-net11-vm1
+--------------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | zone1 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | D2odvQwgqBgH |
| config_drive | |
| created | 2015-06-19T14:49:20Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | ba3fa4f4-46b5-47c8-9d15-3a35e4904a68 |
| image | cirros-0.3.3 (be11b03e-4838-43d2-997d-ebf768b41b1b) |
| key_name | - |
| metadata | {} |
| name | ds47-net11-vm1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | be95bac2e0054234b7a048ed2a33b21d |
| updated | 2015-06-19T14:49:21Z |
| user_id | 3128db857ad549e78e14e2ac140a35bd |
+--------------------------------------+-----------------------------------------------------+
[root@bxb-ds-46 ~]# nova list
+--------------------------------------+----------------+--------+------------+-------------+------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+------------------------------+
| ba3fa4f4-46b5-47c8-9d15-3a35e4904a68 | ds47-net11-vm1 | BUILD | - | NOSTATE | |
+--------------------------------------+----------------+--------+------------+-------------+------------------------------+
[root@bxb-ds-46 ~]# nova show ds47-net11-vm1
+--------------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | zone1 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-06-19T14:49:20Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | ba3fa4f4-46b5-47c8-9d15-3a35e4904a68 |
| image | cirros-0.3.3 (be11b03e-4838-43d2-997d-ebf768b41b1b) |
| key_name | - |
| metadata | {} |
| name | ds47-net11-vm1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| status | BUILD |
| tenant_id | be95bac2e0054234b7a048ed2a33b21d |
| updated | 2015-06-19T14:49:21Z |
| user_id | 3128db857ad549e78e14e2ac140a35bd |
+--------------------------------------+-----------------------------------------------------+