Instance state is not set to ERROR when guestfs file-injection fails

[Tim Miller]

Instances have been observed to remain stuck forever in "BUILD" state, with no errors surfaced to `nova show` after nova boot fails with the following guestfs error.

2013-03-20 18:49:08,590.590 ERROR nova.compute.manager [req-f85ccdcd-74f1-4f50-98eb-b68fb8dc8e1a dba071d520c9438ab9fb91077b6f3248 1ba6328ea66c4041bfab7cfcbc2305cf] [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] Instance failed to spawn
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] Traceback (most recent call last):
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1055, in _spawn
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] block_device_info)
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] admin_pass=admin_password)
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1913, in _create_image
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] instance=instance)
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] self.gen.next()
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1908, in _create_image
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] mandatory=('files',))
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/local/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 304, in inject_data
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] fs.setup()
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] File "/usr/local/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 114, in setup
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] {'imgfile': self.imgfile, 'e': e})
2013-03-20 18:49:08,590.590 14139 TRACE nova.compute.manager [instance: 5f3fe8ba-a148-48e5-8e19-d2f65968b2db] NovaException: Error mounting /var/lib/nova/instances/5f3fe8ba-a148-48e5-8e19-d2f65968b2db/disk with libguestfs (cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH (search path: /usr/lib/guestfs))

[Russell Bryant (russellb)]

What version of nova?

[Tim Miller (tim-miller-0)]

This was observed on nova trunk at git SHA 412b9e15cb28336635c69ebc9e4b381bfa16f9b0 .

[Chuck Short (zulcss)]

Which distro?
Which version of libguestfs?

[Tim Miller (tim-miller-0)]

This was observed on Ubuntu Quantal, using the default version of libguestfs provided by the Ubuntu package repo.

The guestfs error itself is documented here:
  https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1086974

I expect that the 'instance state not set to ERROR' behavior could be reproduced by causing an arbitrary exception to be raised in the vicinity of the stack trace above.

[Joe Gordon (jogo)]

Is this bug still valid? This is a fairly old bug.

[Sahid Orentino (sahid-ferdjaoui)]

We should fallback to localfs if something wrong happens when initialize guestfs. like a setup error

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120467
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6a0e6209cab2ced467b63ce9ce69a41daae669fe
Submitter: Jenkins
Branch: master

commit 6a0e6209cab2ced467b63ce9ce69a41daae669fe
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Wed Sep 10 16:05:22 2014 +0000

    disk/vfs: ensure guestfs capabilities

    Ensures that guestfs is available and well configured. Also
    when checking for capabilities this commit introduces an in-memory
    flag to avoid repeating the process if succeeded.
    If we are able to load guestfs but something wrong happens a fatal
    error is raised.

    DocImpact: We should add a not for ubuntu like systems that use
    libguestfs to execute the command 'update-guestfs-appliance' for
    configuring it.

    Closes-Bug: #1275267
    Closes-Bug: #1157922
    Change-Id: I97b3a23829ea1f3aadfe08ca6448b35010d2f312