nova-compute with IronicDriver failed to start if ironic-api service not started

[Zhenzan Zhou]

When running devstack stack.sh, nova-compute started at 2015-03-11 10:24:01.323, but ironic-api started at 2015-03-11 10:27:04.685. Nova-compute exited at 2015-03-11 10:26:00.083 with NovaException:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
    listener.cb(fileno)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/service.py", line 491, in run_service
    service.start()
  File "/opt/stack/nova/nova/service.py", line 164, in start
    self.manager.init_host()
  File "/opt/stack/nova/nova/compute/manager.py", line 1201, in init_host
    self._destroy_evacuated_instances(context)
  File "/opt/stack/nova/nova/compute/manager.py", line 737, in _destroy_evacuated_instances
    local_instances = self._get_instances_on_driver(context, filters)
  File "/opt/stack/nova/nova/compute/manager.py", line 700, in _get_instances_on_driver
    driver_uuids = self.driver.list_instance_uuids()
  File "/opt/stack/nova/nova/virt/ironic/driver.py", line 422, in list_instance_uuids
    limit=0)
  File "/opt/stack/nova/nova/virt/ironic/client_wrapper.py", line 142, in call
    raise exception.NovaException(msg)
NovaException: Error contacting Ironic server for 'node.list'. Attempt 60 of 60

It doesn't make sense for a service.

[Eric Xie (mark-xiett)]

I use Juno version, 2014.2.
Check the status of nova-compute service after stop ironic-api service, and get it:
# systemctl status openstack-nova-compute.service
openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled)
   Active: active (running) since Wed 2015-03-11 12:01:22 CST; 1h 41min ago
 Main PID: 28778 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           └─28778 /usr/bin/python /usr/bin/nova-compute

Mar 11 12:01:22 controller systemd[1]: Started OpenStack Nova Compute Server.

But in the log file of nova-compute:
2015-03-11 13:40:56.502 28778 TRACE nova.openstack.common.periodic_task NovaException: Error contacting Ironic server for 'node.list'. Attempt 60 of 60

Then start ironic-api service, and the exception was gone.

IMHO this flow does not process properly. When ironic-api service stopped, nova-compute service should be exited. But now it was running. End-user can launch one instance because 'nova-compute' service is up.

[Zhenzan Zhou (zhenzan-zhou)]

I tried to stop ironic-api while nova-compute is running and I reproduced the same behavior as you pasted. For this bug, it only happened at the start up phase.
Personally I don't think it's a right behavior to just shutdown nova-compute if ironic-api service stopped. You can always launch an instance even if nova-compute is stopped. And you will get error like:
"message": "No valid host was found. There are not enough hosts available."
I also tried to just stop ironic-api but keeps nova-compute running, the same error for the instance:
"message": "No valid host was found. There are not enough hosts available."
So it's better to keep nova-compute and once ironic-api is back, it can recover immediately.

BTW, what information do you want to change the status back to normal from incomplete? Thanks.

[Eric Xie (mark-xiett)]

I tried it as below steps:
1) stop ironic-api
2) stop nova-compute
3) start nova-compute
And found that nova-compute was still running.
Anything wrong?

[Zhenzan Zhou (zhenzan-zhou)]

Does it run for long time? Do you see it reports exception periodically after step 3? If the answer is yes, then we probably meet a regression. Please try the latest Kilo code. Thanks.

[Eric Xie (mark-xiett)]

Ok. I will try it as soon as i can:)

[Sean Dague (sdague)]

I do believe that nova compute should not die under these situations.

[Zhenzan Zhou (zhenzan-zhou)]

If I killed libvirtd and then start nova-compute, it got an exception:

HypervisorUnavailable: Connection to the hypervisor is broken on host

and quit immediately. If killed libvirtd while nova-compute is running, it changed compute service status to disabled and won't exit with HypervisorUnavailable exceptions:

2015-04-14 14:57:38.699 DEBUG nova.virt.libvirt.driver [req-5f28fc21-9769-4cd9-8d96-0ee6fd15fd8c None None] Updating compute service status to disabled from (pid=1344) _set_host_enabled /opt/stack/nova/nova/virt/libvirt/driver.py:3021

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/173681

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/173681
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cce06a1e9855d9eed3f7c653200853f23466d791
Submitter: Jenkins
Branch: master

commit cce06a1e9855d9eed3f7c653200853f23466d791
Author: Zhenzan Zhou <email address hidden>
Date: Wed Apr 15 13:27:51 2015 +0800

    Bypass ironic server not available issue

    The ironic driver needs enhancement for exception handling.
    This patch is a workaround to make devstack with ironic enabled
    success. A more elegant patch should be made later in ironic
    driver for exception handling.

    Change-Id: Ibace25ad905a8278ecea4b02c69c59737a490d3a
    Closes-Bug: #1430616