task_state is not 'None' when vm stay in 'ERROR' state

[Rui Chen]

Compute instance task states represent what is happening to the instance at the current moment. When the instance is 'ERROR' state, the 'spawning' task_state make users confused, 'spawning' job had been done and failed, it should been set to 'None'.

1. Version of Nova

$ git log -1
commit 4cf6ef68199183697a0209751575f88fe5b2a733
Merge: f40619b 70ba331
Author: Jenkins <email address hidden>
Date: Wed May 27 22:14:34 2015 +0000

    Merge "improve speed of some ec2 keypair tests"

2. Log files

stack@devstack:/home/devstack/logs$ [master]$ nova list
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------------------------+
| 5fa49075-f0a0-4806-bdf3-0cedd09c7c6f | chenrui_again | ERROR | spawning | NOSTATE | |
| 19920850-86b0-4904-8431-bf1ed6f9cea7 | chenrui_vm | ACTIVE | - | Running | private=fd6b:c8ae:7d0d:0:f816:3eff:fe96:bbfa, 10.0.0.3 |
+--------------------------------------+---------------+--------+------------+-------------+--------------------------------------------------------+

2015-05-28 10:42:14.618 4705 WARNING nova.network.neutronv2.api [-] [instance: 5fa49075-f0a0-4806-bdf3-0cedd09c7c6f] Neutron error: No more fixed IPs in network: ecf5d5d3-6198-4c95-84d4-db633fb09526
2015-05-28 10:42:14.619 4705 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager Traceback (most recent call last):
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager File "/opt/stack/nova/nova/compute/manager.py", line 1535, in _allocate_network_async
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager dhcp_options=dhcp_options)
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager File "/opt/stack/nova/nova/network/neutronv2/api.py", line 667, in allocate_for_instance
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager File "/opt/stack/nova/nova/network/neutronv2/api.py", line 659, in allocate_for_instance
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager File "/opt/stack/nova/nova/network/neutronv2/api.py", line 321, in _create_port
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager raise exception.NoMoreFixedIps(net=network_id)
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager NoMoreFixedIps: No fixed IP addresses available for network: ecf5d5d3-6198-4c95-84d4-db633fb09526
2015-05-28 10:42:14.619 4705 TRACE nova.compute.manager
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
    listener.cb(fileno)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 1552, in _allocate_network_async
    six.reraise(*exc_info)
  File "/opt/stack/nova/nova/compute/manager.py", line 1535, in _allocate_network_async
    dhcp_options=dhcp_options)
  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 667, in allocate_for_instance
    self._delete_ports(neutron, instance, created_port_ids)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 659, in allocate_for_instance
    security_group_ids, available_macs, dhcp_opts)
  File "/opt/stack/nova/nova/network/neutronv2/api.py", line 321, in _create_port
    raise exception.NoMoreFixedIps(net=network_id)
NoMoreFixedIps: No fixed IP addresses available for network: ecf5d5d3-6198-4c95-84d4-db633fb09526
Removing descriptor: 19

3. Reproduce steps:

* create a neutron network and subnet, there is only 1 fixed ip in the subnet.
   neutron subnet-create --allocation-pool start=100.100.1.5,end=100.100.1.5 ecf5d5d3-6198-4c95-84d4-db633fb09526 100.100.1.1/24
* boot a instance with the network_id.

Expected result:
* booting failed, instance is 'ERROR' state and task_state is 'None'

Actual result:
* booting failed, instance is 'ERROR' state and task_state is 'spawning'

[jichenjc (jichenjc)]

guess it's because NetworkInfoAsyncWrapper is async so the reverts_task_state is not helpful

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/187145

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/187145
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=67f29b196c3bbbde413c9176b83c8b878753a38d
Submitter: Jenkins
Branch: master

commit 67f29b196c3bbbde413c9176b83c8b878753a38d
Author: Rui Chen <email address hidden>
Date: Mon Jun 1 17:20:42 2015 +0800

    Set task_state=None when booting instance failed

    When booting instance failed in nova-compute, the
    instance is ERROR state, but the instance.task_state
    is not None. According to the definition of task_state,
    instance task states represent what is happening to
    the instance at the current moment. The users will be confused
    when they find a instance with vm_state=ERROR and task_state=
    spawning.

    Change-Id: I6e8d1bd549052317a8b60052e2a0ffbf800af13f
    Closes-Bug: #1459496