bad configuration for libguestfs

[Sahid Orentino]

An error is reported by guestfs when exercising but hidden by an except reported in a log level DEBUG:

  http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/disk/api.py#n214

In Nova a clean is in progress on this part of code; We normally would like to break the process if something goes wrong at this step. However the CI failed since the except has been removed:

  https://review.openstack.org/#/c/119104/18/nova/virt/disk/api.py,cm

2015-01-15 15:19:35.840 28636 ERROR nova.compute.manager [-] [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] Instance failed to spawn
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] Traceback (most recent call last):
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/compute/manager.py", line 2290, in _build_resources
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] yield resources
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/compute/manager.py", line 2160, in _build_and_run_instance
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] flavor=flavor)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2391, in spawn
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] admin_pass=admin_password)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2790, in _create_image
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] project_id=instance['project_id'])
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 201, in cache
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] *args, **kwargs)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 478, in create_image
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] copy_qcow2_image(base, self.path, size)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] return f(*args, **kwargs)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 444, in copy_qcow2_image
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] disk.extend(target, size, use_cow=True)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/disk/api.py", line 167, in extend
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] if not is_image_extendable(image, use_cow):
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/disk/api.py", line 217, in is_image_extendable
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] fs = vfs.VFS.instance_for_image(image, 'qcow2', None)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/disk/vfs/api.py", line 57, in instance_for_image
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] vfs.inspect_capabilities()
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] File "/opt/stack/new/nova/nova/virt/disk/vfs/guestfs.py", line 82, in inspect_capabilities
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] _("libguestfs installed but not usable (%s)") % e)
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202] NovaException: libguestfs installed but not usable (cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH (search path: /usr/lib/guestfs))
2015-01-15 15:19:35.840 28636 TRACE nova.compute.manager [instance: 162caeb6-5a8e-413f-8df2-300cf4e66202]

We should correctly configure libguestfs as indicated in the backtrace.

[Richard Jones (rjones-redhat)]

What host OS? What version of libguestfs? Where did you obtain libguestfs from?

[Sahid Orentino (sahid-ferdjaoui)]

This comes from our CI installed with devstack, I do not have a lot of information about it. I guess it runs on Ubuntu.

You can get more information of the error here:

   http://logs.openstack.org/04/119104/18/check/check-devstack-dsvm-cells/bf6871b/logs/screen-n-cpu.txt.gz

Or more information about the environment running here:

    http://logs.openstack.org/04/119104/18/check/check-devstack-dsvm-cells/bf6871b/logs/

From my understand we do not install the package 'update-guestfs-appliance' - I have tried to make this package installed but it takes a lot of time and was not accepted:

    https://review.openstack.org/#/c/79577/

[Silvan Kaiser (2-silvan)]

Had this occur in some manual testing on our CI system today. host OS Ubuntu 14.04 . Libguestfs package was not installed (which is weird as i'd expect devstack to do this...).

[Silvan Kaiser (2-silvan)]

Correction: libguestfs-tools package version was < 1.24.5-1 , the latter is the version i updated to during debugging.

[Roman Podoliaka (rpodolyaka)]

The problem on the Nova side is that it only does a fall back to another backend, if `guestfs` Python module can't be imported: https://github.com/openstack/nova/blob/91f8cc9c153b61a5aed081c2d1b44b21f35d3311/nova/virt/disk/vfs/api.py#L42-L73

At the same time, if `guestfs` Python-binding is installed and importable, it can still be unusable, what we see in the bug message: e.g. when an appliance hasn't been built properly.

IMO, we should fix that in Nova and gracefully fall back to NBD, if guestfs is not usable.

[Roman Podoliaka (rpodolyaka)]

From the configuration perspective, there are a few problems with guestfs in Ubuntu 14.04:

1) libguestfs0 lib is installed, but python binding is not by devstack by default (unless you also install Ironic - https://github.com/openstack-dev/devstack/blob/5751017ae3c9150eac03f03257f45ca2356b8484/files/debs/ironic#L11)

2) libguestfs0 comes without a built appliance - so guestfs is not really usable

3) libguestfs-tools is not installed, but required to built an appliance

4) after you install libguestfs-tools manually, you need to also build an appliance by the means of /usr/sbin/update-guestfs-appliance

5) even with the appliance guestfs will fail with:

/usr/bin/supermin-helper: open: /boot/vmlinuz-3.13.0-61-generic: Permission denied

quote from libguestfs FAQ (http://libguestfs.org/guestfs-faq.1.html):

Ubuntu
We don't have a full time Ubuntu maintainer, and the packages supplied by Canonical (which are outside our control) are sometimes broken.

Canonical decided to change the permissions on the kernel so that it's not readable except by root. This is completely stupid, but they won't change it (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/759725). So every user should do this:

 sudo chmod 0644 /boot/vmlinuz*

[Roman Podoliaka (rpodolyaka)]

^ I'm not sure Devstack should really do all that, but we could make it configurable, so that we actually test Nova/libguestfs cooperation in the gate

[Markus Zoeller (markus_z) (mzoeller)]

Duplicates to this bug:
bug 1507915 "libguestfs launch image failed in ubuntu"
bug 1507016 "openstack nova dont boot with iso and attached volume with block-device"

[Markus Zoeller (markus_z) (mzoeller)]

Ignore my last comment, the duplicates can be seen in the top right corner...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/237547

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237547
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=92ae0f1077e4c5916d99777b032aaf0840e7ab93
Submitter: Jenkins
Branch: master

commit 92ae0f1077e4c5916d99777b032aaf0840e7ab93
Author: Chung Chih, Hung <email address hidden>
Date: Tue Oct 20 12:41:46 2015 +0000

    libvirt - Add log if libguestfs can't read host kernel

    Host's kernel only allows a root user to have read/write permission in
    ubuntu. If compute-service didn't have read permission then libguestfs
    will launch image fail.

    In libguestfs offical FAQ site had point out this issue, following is
    the link
    http://libguestfs.org/guestfs-faq.1.html#binaries
    It had suggested users to change host's kernel permission. But this
    action should be handled by other patch. Here only give a hint what's
    going wrong.

    Change-Id: I36c93610831e2935d46f7ee37f95619fe6dc1481
    Related-Bug: 1413142
    Closes-Bug: 1491216

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/337091

[Kevin Zhao (kevin-zhao)]

Hi All,
    I find the merged patch above(https://review.openstack.org/237547) may has a mistake for judgement of os.access("/boot/vmlinuz-%s" % os.uname()[2], os.R_OK).
Reproduce:
1. export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
2. When I running injecting files into image, got error:

 File "/opt/stack/nova/nova/virt/disk/vfs/guestfs.py", line 85, in inspect_capabilities
2016-07-04 09:17:40.816 TRACE nova.compute.manager [instance: b19dac5b-ac49-4a1c-8f88-baaa91565867] _("libguestfs installed but not usable (%s)") % e)
2016-07-04 09:17:40.816 TRACE nova.compute.manager [instance: b19dac5b-ac49-4a1c-8f88-baaa91565867] NovaException: libguestfs installed but not usable (/usr/bin/supermin exited with error status 1, see debug messages above)

And this for detailed in libguestfs:
supermin: kernel: picked kernel vmlinuz-4.4.0-28-generic
cp: cannot open '/boot/vmlinuz-4.4.0-28-generic' for reading: Permission denied
supermin: cp -p '/boot/vmlinuz-4.4.0-28-generic' '/var/tmp/.guestfs-1002/appliance.d.kxo2ix12/kernel': command failed, see earlier errors
libguestfs: trace: launch = -1 (error)
2016-07-04 09:17:40.815 ERROR nova.virt.libvirt.driver [req-5915d6e6-5cfe-419f-bfb0-f5ba4d80e2aa admin admin] [instance: b19dac5b-ac49-4a1c-8f88-baaa91565867] Error injecting data into image 289f8d67-f641-4ba0-b15a-7a0b7c99942a (libguestfs installed but not usable (/usr/bin/supermin exited with error status 1, see debug messages above))

It is clearly that I libguestfs don't have the permission for reading the kernel. But I can't get the error information from exception.LibguestfsCannotReadKernel().
    So I modify the code here for enabling the report : https://review.openstack.org/#/c/337091/
    All this detailed n-cpu log is in the attachment.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Zhao (<email address hidden>) on branch: master
Review: https://review.openstack.org/337091

[Sean Dague (sdague)]

Andreaf fixed this this week

[Ihar Hrachyshka (ihar-hrachyshka)]

For the record, the fix Sean mentioned is: https://review.openstack.org/#/c/404981/ I am going to backport it to Newton because it fails there too.

[Ihar Hrachyshka (ihar-hrachyshka)]

Oh nevermind, it's already backported as https://review.openstack.org/#/c/476368/