Nova-compute fails if it doesn't wait for libvirt-bin

[Jian Wen]

If Nova-compute tries to start before libvirt-bin is up it will fail.

2013-10-08 07:24:16.703 2537 INFO nova.manager [-] Skipping periodic task _periodic_update_dns because its interval is negative
2013-10-08 07:24:18.683 2537 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2013-10-08 07:24:19.243 INFO nova.openstack.common.rpc.common [req-d1fb1b3c-6666-43db-b421-ac67c1c956b8 None None] Connected to AMQP server on *.example.com:5672
2013-10-08 07:24:19.281 2537 AUDIT nova.service [-] Starting compute node (version 2013.1.3)
2013-10-08 07:24:22.129 INFO nova.compute.manager [req-660b8846-b31c-405c-b37b-a92273db35a2 None None] Updating host status
2013-10-08 07:24:22.654 2537 CRITICAL nova [-] End of file while reading data: Input/output error
2013-10-08 07:24:22.654 2537 TRACE nova Traceback (most recent call last):
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/bin/nova-compute", line 85, in <module>
2013-10-08 07:24:22.654 2537 TRACE nova service.wait()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 689, in wait
2013-10-08 07:24:22.654 2537 TRACE nova _launcher.wait()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 209, in wait
2013-10-08 07:24:22.654 2537 TRACE nova super(ServiceLauncher, self).wait()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 179, in wait
2013-10-08 07:24:22.654 2537 TRACE nova service.wait()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2013-10-08 07:24:22.654 2537 TRACE nova return self._exit_event.wait()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-10-08 07:24:22.654 2537 TRACE nova return hubs.get_hub().switch()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2013-10-08 07:24:22.654 2537 TRACE nova return self.greenlet.switch()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2013-10-08 07:24:22.654 2537 TRACE nova result = function(*args, **kwargs)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 147, in run_server
2013-10-08 07:24:22.654 2537 TRACE nova server.start()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 429, in start
2013-10-08 07:24:22.654 2537 TRACE nova self.manager.init_host()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 620, in init_host
2013-10-08 07:24:22.654 2537 TRACE nova self._report_driver_status(context)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3744, in _report_driver_status
2013-10-08 07:24:22.654 2537 TRACE nova capabilities = self.driver.get_host_stats(refresh=True)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3449, in get_host_stats
2013-10-08 07:24:22.654 2537 TRACE nova return self.host_state.get_host_stats(refresh=refresh)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3449, in get_host_stats
2013-10-08 07:24:22.654 2537 TRACE nova return self.host_state.get_host_stats(refresh=refresh)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3771, in get_host_stats
2013-10-08 07:24:22.654 2537 TRACE nova self.update_status()
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3787, in update_status
2013-10-08 07:24:22.654 2537 TRACE nova self.driver.get_memory_mb_used())
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2631, in get_memory_mb_used
2013-10-08 07:24:22.654 2537 TRACE nova return self.get_memory_mb_total() - avail / 1024
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2542, in get_memory_mb_total
2013-10-08 07:24:22.654 2537 TRACE nova return self._conn.getInfo()[1]
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-10-08 07:24:22.654 2537 TRACE nova result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-10-08 07:24:22.654 2537 TRACE nova rv = execute(f,*args,**kwargs)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-10-08 07:24:22.654 2537 TRACE nova rv = meth(*args,**kwargs)
2013-10-08 07:24:22.654 2537 TRACE nova File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3450, in getInfo
2013-10-08 07:24:22.654 2537 TRACE nova if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
2013-10-08 07:24:22.654 2537 TRACE nova libvirtError: End of file while reading data: Input/output error
2013-10-08 07:24:22.654 2537 TRACE nova

[Jian Wen (wenjianhn)]

# TODO(wenjianhn):
1. steps to reproduce the bug
 2. steps to verify the fix

[Jian Wen (wenjianhn)]

Steps to reproduce the bug:

On a new Ubuntu 12.04 server, execute the following commands:
  sudo apt-get install -y python-software-properties
  sudo add-apt-repository cloud-archive:grizzly
  sudo apt-get update
  sudo apt-get install -y nova-compute

  sudo stop nova-compute
  sudo stop libvirt-bin

  # simulate that libvirt-bin starts after nova-compute is running
  sudo start nova-compute
  sleep 5

   sudo status nova-compute
  # expected result:
  # start/running
  # actual result:
  # stop/waiting

[Jian Wen (wenjianhn)]

Steps to verify the fix:

1. Copy the patched upstart configuration to /etc/init
2. Execute the following commands:

   sudo stop nova-compute
   sudo stop libvirt-bin

   sudo start nova-compute
   sleep 10
   sudo status nova-compute
   # nova-compute start/running, process 13692
   # It's up and running. No error log is appended in nova-compute.log.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debian/nova-compute.upstart: Wait for libvirt-bin if needed. (LP: #1261335)" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]