Ubuntu

Error: unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: No such file or directory

Reported by Paul Belanger on 2011-12-21
50
This bug affects 7 people
Affects Status Importance Assigned to Milestone
ubuntu-cloud-archive
Undecided
Unassigned
nova (Ubuntu)
Status tracked in Trusty
Raring
Medium
Chris J Arges
Saucy
Medium
Chris J Arges
Trusty
Medium
James Page

Bug Description

SRU Justification:
[Impact]
* If nova-compute tries to start before libvirt-bin is up it will fail. And result in the above error.

[Test Case]
* Execute the following commands:
  sudo apt-get install -y python-software-properties
  sudo add-apt-repository cloud-archive:grizzly
  sudo apt-get update
  sudo apt-get install -y nova-compute

  sudo stop nova-compute
  sudo stop libvirt-bin

  # simulate that libvirt-bin starts after nova-compute is running
  sudo start nova-compute
  sleep 5

   sudo status nova-compute
  # expected result:
  # start/running
  # actual result:
  # stop/waiting

[Regression Potential]
* This is a modification to the upstart script, and is currently commited in icehouse.

--

It looks like there is a race condition between nova-compute-kvm and libvirt. We should move the init script into nova-compute-kvm and add a dependency on libvirt. I'll see about updating the package to handle this

---

2011-12-20 21:56:47,255 CRITICAL nova [-] unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: No such file or directory
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-compute", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/service.py", line 357, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/service.py", line 107, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/service.py", line 77, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/service.py", line 149, in start
(nova): TRACE: self.manager.update_available_resource(ctxt)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 129, in wrapped
(nova): TRACE: raise Error(str(e))
(nova): TRACE: Error: unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: No such file or directory
(nova): TRACE:

Paul Belanger (pabelanger) wrote :

I was mistaken about breaking out upstart into other packages, both use libvirt-bin. See attached patch.

Paul Belanger (pabelanger) wrote :

Opps, wrong patch. This is the one

Thierry Carrez (ttx) wrote :

This is actually an Ubuntu packaging issue.

affects: nova → ubuntu
affects: ubuntu → nova (Ubuntu)

The attachment "nova-compute.upstart.diff" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Serge Hallyn (serge-hallyn) wrote :

If the upstart job is going to wait on libvirt-bin to start anyway, can the other two conditions not be dropped as they are implied by libvirt-bin (which itself waits on runlevel 2)?

Changed in nova (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Serge Hallyn (serge-hallyn) wrote :

(setting to confirmed as the race certainly is possible based on the 'start on's)

Paul Belanger (pabelanger) wrote :

No, I believe we need to keep it. Since nova is not explicitly dependent on libvirt.

Paul Belanger (pabelanger) wrote :

Debdiff attached

Paul Belanger (pabelanger) wrote :

v2: Based on SpamapS comments from #ubuntu-server

devendra (devedevendra) on 2012-03-13
Changed in nova (Ubuntu):
assignee: nobody → devendra (devedevendra)
assignee: devendra (devedevendra) → nobody
James Page (james-page) wrote :

Hi Paul

Thanks for you work on this bug.

I understand what this bug is trying to resolve but the proposed patch to the upstart configuration will break the nova-compute-xcp package which provides nova-compute-hypervisor but does not depend on libvirt-bin.

In this type of deployment a nova-compute node will never start...

Un-subscribing ubuntu-sponsors for the time being.

Please re-subscribe this team when you feel that you have a suitable patch - or even better propose a merge against lp:~ubuntu-server-dev/nova/essex!

Thanks

James

I've been bitten by this bug too.

> Please re-subscribe this team when you feel that you have a suitable patch

Any hint about what would be the proper way of fixing it?

James Troup (elmo) on 2012-12-19
tags: added: canonistack
James Page (james-page) wrote :

I have a way todo this now; I'll push in with the next set of icehouse updates I do.

Changed in nova (Ubuntu):
status: Triaged → In Progress
assignee: nobody → James Page (james-page)
James Page (james-page) on 2013-12-18
Changed in nova (Ubuntu Trusty):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in nova (Ubuntu Raring):
status: New → Confirmed
Changed in nova (Ubuntu Saucy):
status: New → Confirmed
Chris J Arges (arges) on 2014-01-09
Changed in nova (Ubuntu Raring):
status: Confirmed → In Progress
importance: Undecided → Medium
Changed in nova (Ubuntu Saucy):
importance: Undecided → Medium
Changed in nova (Ubuntu Raring):
assignee: nobody → Chris J Arges (arges)
Changed in nova (Ubuntu Saucy):
status: Confirmed → In Progress
assignee: nobody → Chris J Arges (arges)
description: updated
Chris J Arges (arges) wrote :

I have linked branches for an SRU for grizzly and horizon. Please review when you get a chance. Thanks!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 1:2014.1~b2-0ubuntu3

---------------
nova (1:2014.1~b2-0ubuntu3) trusty; urgency=medium

  * debian/tests/nova-compute-daemons: Drop nova-compute-uml.
 -- Chuck Short <email address hidden> Fri, 31 Jan 2014 14:40:23 -0500

Changed in nova (Ubuntu Trusty):
status: Fix Committed → Fix Released
James Page (james-page) on 2014-03-10
Changed in nova (Ubuntu Raring):
status: In Progress → Won't Fix
James Page (james-page) on 2014-03-10
Changed in cloud-archive:
status: New → In Progress

Hello Paul, or anyone else affected,

Accepted nova into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/1:2013.2.2-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Saucy):
status: In Progress → Fix Committed
tags: added: verification-needed
James Page (james-page) wrote :

Tested with proposed; starting nova-compute always ensures that libvirt is started first.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 1:2013.2.2-0ubuntu1.1

---------------
nova (1:2013.2.2-0ubuntu1.1) saucy-proposed; urgency=low

  [ Jian Wen ]
  * d/nova-compute.upstart: If libvirt-bin is installed, then wait
    for it to reach running state before starting nova-compute
    (LP: #907152, #1190280)
 -- Chris J Arges <email address hidden> Thu, 09 Jan 2014 12:30:55 -0600

Changed in nova (Ubuntu Saucy):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers