libguestfs launch image failed in ubuntu

[Chung Chih, Hung]

I had following settings when I want to enable inject feature in nova.
[libvirt]
inject_partition = -1
inject_key = True

But nova-compute service will raise following exception
2015-10-20 07:12:57.318 ERROR nova.virt.libvirt.driver [req-777e2fe1-a4f0-4c16-bb71-ee34e59aa2ba admin admin] [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] Error injecting data into image 865c98b5-ffd7-4f32-b1af-07b273fcc07d (libguestfs installed but not usable (/usr/bin/supermin-helper exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs))
2015-10-20 07:12:57.319 ERROR nova.compute.manager [req-777e2fe1-a4f0-4c16-bb71-ee34e59aa2ba admin admin] [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] Instance failed to spawn
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] Traceback (most recent call last):
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/compute/manager.py", line 2172, in _build_resources
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] yield resources
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/compute/manager.py", line 2019, in _build_and_run_instance
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] block_device_info=block_device_info)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2437, in spawn
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] admin_pass=admin_password)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2969, in _create_image
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] instance, network_info, admin_pass, files, suffix)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2778, in _inject_data
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] instance=instance)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] six.reraise(self.type_, self.value, self.tb)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2772, in _inject_data
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] mandatory=('files',))
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/disk/api.py", line 414, in inject_data
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] fs = vfs.VFS.instance_for_image(image, partition)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/disk/vfs/api.py", line 62, in instance_for_image
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] vfs.inspect_capabilities()
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] File "/opt/stack/nova/nova/virt/disk/vfs/guestfs.py", line 89, in inspect_capabilities
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] _("libguestfs installed but not usable (%s)") % e)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] NovaException: libguestfs installed but not usable (/usr/bin/supermin-helper exited with error status 1.
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] To see full error messages you may need to enable debugging.
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs)
2015-10-20 07:12:57.319 TRACE nova.compute.manager [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb]
2015-10-20 07:12:57.322 INFO nova.compute.manager [req-777e2fe1-a4f0-4c16-bb71-ee34e59aa2ba admin admin] [instance: b2d39549-96d5-42c6-bdab-a0cee10b24bb] Terminating instance

Why guestfs will inspect capabilities fail?
Because of host's kernel only allow root user had read/write permission.
If compute-service didn't had read permission then it will launch image fail.
In libguestfs offical FAQ site had point out this issue, following is the link
http://libguestfs.org/guestfs-faq.1.html#binaries
It had suggested users to change host's kernel permission.

You can also check result by guestfish command:
> export LIBGUESTFS_DEBUG=1
> export LIBGUESTFS_TRACE=1
> guestfish -a /dev/null
<fs> launch
...
...
/usr/bin/supermin-helper: open: /boot/vmlinuz-3.13.0-55-generic: Permission denied
libguestfs: error: /usr/bin/supermin-helper exited with error status 1, see debug messages above
...
...

We can have three way to resolve this problem.
1. Open service in root permission
2. Change kernel's permission in compute-service
3. Check whether service had permission to read kernel. Suggest users to modify permission instead directly modify permission. Then users need to manually change kernel's permission.

We shouldn't open service in root permission, therefore first way shouldn't been accepted.
It will probably have security issue if service can directly change file's permission.
At last, I prefer third way.
Because of this issue will only happen in ubuntu os and previous reasons.

libguestfs-tools 1:1.24.5-1

[Markus Zoeller (markus_z) (mzoeller)]

@lyanchih:

I tend to believe that this is a duplicate to bug 1413142 so I close this bug as duplicate so that the effort to solve this issue is solely focues on bug 1413142.

[Richard Jones (rjones-redhat)]

Ubuntu has broken kernel permissions. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/759725

[Chris Martin (6-chris-z)]

I think the duplicate status is wrong. @markus_z / @mzoeller: I think bug 1413142 is about Nova _failing to report_ when libguestfs could not read the kernel (which may have been fixed). This bug is about the broader problem: the Nova features which rely on libguestfs do not work out of the box when Nova is installed on the most widely used distro(?) for OpenStack.

One thing that fails is having libvirt inspect which partition an SSH key or admin password should be injected into ("inject_partition = -1" in the "libvirt" section of nova.conf).

It sounds like the Ubuntu maintainers are unlikely to change their default file permissions on compressed kernels, so is it possible for Nova to compensate for this, perhaps by modifying the rootwrap so that guestfs or /usr/bin/supermin is run with root privileges? Would that be reasonable?

Otherwise, it seems that everyone who deploys Nova (including projects like OpenStack-Ansible and Fuel) will continue to trip over this. Maybe we'll implement workarounds (like modifying Nova's rootwrap ourselves or just making the kernel readable to unprivileged users), but it'd be really nice to fix this in Nova.

[Chris Martin (6-chris-z)]

After digging deeper I'm now unsure if rootwrap is a practical solution. Apparently "nova.utils.execute(run_as_root=True)" is used to request a command to be passed through rootwrap -- but the code running the commands (python-guestfs) is actually a system-level Python package which imports a C library. Nova just imports the guestfs Python package and uses its API. It doesn't appear that Nova has the opportunity to execute anything using rootwrap -- the guestfs code itself would need to be modified.

In OpenStack-Ansible I have proposed the workaround of just modifying file permissions on the kernel:
https://review.openstack.org/#/c/451971/

This all works once the kernel is made readable to the nova user.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/451971
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=2bd15db0364f225b5af39e2b98e1c12cb75e52d4
Submitter: Jenkins
Branch: master

commit 2bd15db0364f225b5af39e2b98e1c12cb75e52d4
Author: cmart <email address hidden>
Date: Thu Mar 30 11:00:55 2017 -0700

    nova user can read kernel for libguestfs on Ubuntu

    Problem: libvirt password/key injection uses libguestfs to mount the
    guest filesystem. libguestfs uses a supermin appliance, and in order to
    create this appliance, libguestfs (running as nova user) must read the
    host's kernel. Unfortunately, Ubuntu sets file permissions which make
    compressed kernels non-readable to non-root users, and this breaks
    libvirt password/key injection on compute hosts running Ubuntu.

    Solution: When compute hosts are running Ubuntu AND the deployer has
    enabled libvirt password or SSH key injection, do the following:
    - Run `dpkg-statoverride` to set file permissions on compressed
      kernel (/boot/vmlinuz-*), readable to group 'nova'
    - Install a script which does same for each new kernel installed via
      system updates in the future

    Related-Bug: #1507915
    Change-Id: Ic96b69bb80ce11001b2ee5d63324a12b0f68456d

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/454269

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/454270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/454269
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=b715834259ea435839ac6f72cdf92b732e928726
Submitter: Jenkins
Branch: stable/ocata

commit b715834259ea435839ac6f72cdf92b732e928726
Author: cmart <email address hidden>
Date: Thu Mar 30 11:00:55 2017 -0700

    nova user can read kernel for libguestfs on Ubuntu

    Problem: libvirt password/key injection uses libguestfs to mount the
    guest filesystem. libguestfs uses a supermin appliance, and in order to
    create this appliance, libguestfs (running as nova user) must read the
    host's kernel. Unfortunately, Ubuntu sets file permissions which make
    compressed kernels non-readable to non-root users, and this breaks
    libvirt password/key injection on compute hosts running Ubuntu.

    Solution: When compute hosts are running Ubuntu AND the deployer has
    enabled libvirt password or SSH key injection, do the following:
    - Run `dpkg-statoverride` to set file permissions on compressed
      kernel (/boot/vmlinuz-*), readable to group 'nova'
    - Install a script which does same for each new kernel installed via
      system updates in the future

    Related-Bug: #1507915
    Change-Id: Ic96b69bb80ce11001b2ee5d63324a12b0f68456d
    (cherry picked from commit 2bd15db0364f225b5af39e2b98e1c12cb75e52d4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/454270
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=c15d491020d857dfb8d8cf819e59a2effa42b2a5
Submitter: Jenkins
Branch: stable/newton

commit c15d491020d857dfb8d8cf819e59a2effa42b2a5
Author: cmart <email address hidden>
Date: Thu Mar 30 11:00:55 2017 -0700

    nova user can read kernel for libguestfs on Ubuntu

    Problem: libvirt password/key injection uses libguestfs to mount the
    guest filesystem. libguestfs uses a supermin appliance, and in order to
    create this appliance, libguestfs (running as nova user) must read the
    host's kernel. Unfortunately, Ubuntu sets file permissions which make
    compressed kernels non-readable to non-root users, and this breaks
    libvirt password/key injection on compute hosts running Ubuntu.

    Solution: When compute hosts are running Ubuntu AND the deployer has
    enabled libvirt password or SSH key injection, do the following:
    - Run `dpkg-statoverride` to set file permissions on compressed
      kernel (/boot/vmlinuz-*), readable to group 'nova'
    - Install a script which does same for each new kernel installed via
      system updates in the future

    Related-Bug: #1507915
    Change-Id: Ic96b69bb80ce11001b2ee5d63324a12b0f68456d
    (cherry picked from commit 2bd15db0364f225b5af39e2b98e1c12cb75e52d4)

[Sean Dague (sdague)]

I think this is basically an upstream Ubuntu bug, and nova force doing this is probably bad form. It's fine for installers to change ubuntu defaults to work around it.

[Launchpad Janitor (janitor)]

[Expired for openstack-ansible because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]