Can't create guests from ISOs on Samba share and NTFS - "search permissions" error

[Fabián Rodríguez]

Binary package hint: virt-manager

When trying to create a new VM using virt-manager, an error is displayed inviting to fix permissions:
"The emulator may not have search permissions for the path '/home/myuser/.gvfs/public on remotehost/ISOs/someubuntuiso.iso'.
Do you want to correct this now ?

If one chooses 'Yes' a final error is displayed which includes:
"Errors were encountered changing permissions for the following directories:
/home/myuser/.gvfs/public on remotehost : [Errno 95] Operation not supported: '/home/myuser/.gvfs/public on remotehost' /home/myuser/.gvfs/public on remotehost/ISOs : [Errno 95] Operation not supported: '/home/myuser/.gvfs/public on remotehost/ISOs'

This seems to be the same as the following upstream bug:
https://bugzilla.redhat.com/show_bug.cgi?id=517304

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: virt-manager (not installed)
ProcVersionSignature: Ubuntu 2.6.35-24.42-generic 2.6.35.8
Uname: Linux 2.6.35-24-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Mon Jan 3 12:48:29 2011
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release i386 (20101007)
ProcEnviron:
 LANG=en_CA.utf8
 SHELL=/bin/bash
SourcePackage: virt-manager

[Tim (tim-redhat-bugs)]

Description of problem:
When trying to install a guest from an ISO file in an NFS-mounted directory I get this from virt-manager:

Error

Unable to complete install '<class 'libvirt.libvirtError'> internal error unable to start guest: libvir: QEMU error : cannot set ownership on /mnt/cyberelk/Fedora-11-x86_64-DVD/Fedora-11-x86_64-DVD.iso: Read-only file system

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/create.py", line 1489, in do_install
    dom = guest.start_install(False, meter = meter)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 628, in start_install
    return self._do_install(consolecb, meter, removeOld, wait)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line 726, in _do_install
    self.domain = self.conn.createLinux(install_xml, 0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1077, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error unable to start guest: libvir: QEMU error : cannot set ownership on /mnt/cyberelk/Fedora-11-x86_64-DVD/Fedora-11-x86_64-DVD.iso: Read-only file system

'

Version-Release number of selected component (if applicable):
virt-manager-0.8.0-1.fc12.noarch
libvirt-0.7.0-3.fc12.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Mount NFS directory with '-ocontext="system_u:object_r:virt_content_t:s0"
2.Install guest from ISO in that directory.

Actual results:
Error message.

Additional info:
$ mount | grep virt
cyberelk:/mnt/archive/rh on /mnt/cyberelk type nfs (rw,context="system_u:object_r:virt_content_t:s0",addr=192.168.2.1)

$ ls -l /mnt/cyberelk/Fedora-11-x86_64-DVD/
total 4172188
-rw-rw-r--. 1 twaugh twaugh 1620 2009-06-17 07:44 Fedora-11-x86_64-CHECKSUM
-rw-rw-r--. 1 root root 4268124160 2009-06-18 00:06 Fedora-11-x86_64-DVD.iso
drwxr-xr-x. 2 root root 4096 2009-07-22 17:04 loop

[Tim (tim-redhat-bugs)]

Hacky workaround: export the NFS directory with 'rw,no_root_squash'.

[Tim (tim-redhat-bugs)]

Actually I'm not so sure that even does work around it. When booting a guest I get:

Booting from CD-Rom...
CDROM boot failure code : 0003
Boot failed: could not read the boot disk

FATAL: No bootable device.

(Of course, the ISO checksum is correct..)

[Daniel (daniel-redhat-bugs)]

A quick summary of what's going on here for the benefit of anyone using NFS

 - libvirt now runs QEMU as a 'qemu:qemu' user/group pair, instead of root:root
 - libvirt will chown() disk images to this user pair when starting a guest to ensure they're accessible (similar to it setting SELinux labels)

Now for the problems you've exposed

 - If the NFS share is read-only, clearly we can't change ownership at all
 - If the NFS share is read-write, but root-squash, a libvirtd won't be able to change ownership because it gets squashed.

So

 - If the disk already has correct ownership, there's nothing for us todo, we should skip this chown step entirely

 - If the guest disk is read-only, we should treat this as non-fatal. The disk quite likely has world-read permission set which is more than sufficient.

 - If the guest disk is read-write, and has incorrect ownership, we have to try and change it, but if it fails there's nothing we can do. The admin will have to fix it.

There is one further problem wrt to NFS and the storage capabilities. Since libvirtd run as root, if you're using a libvirt storage pool based on NFS then its neccessary to have root squash disabled at this time

We need to figure out a way to let libvirt create files on NFS storage pools with the correct UID right from start, so that we don't ever need to chown things. Since libvirtd runs as root, this probably means we need to spawn a small helper program to create NFS files, which can change its UID to the desired target, avoiding the root squash issue.

In the mean time, there are two workarounds:

 - Use a writable NFS export with root squash disabled

Or

 - Edit /etc/libvirt/qemu.conf and set it to run guest as user="root" and group="root", instead of qemu/qemu (ie reverting to < F11 behaviour)

The choice is yours based on whether security of host OS more important (change the NFS server), or security of NFS server more important (change QEMU to run as root).

[Mark (mark-redhat-bugs)]

dan: should we split some of this up into separate bugs?

[Tim (tim-redhat-bugs)]

Neither work-around works for me actually.

Use writable NFS export with no_root_squash: see comment #2

Run guest as root:root: see bug #517619.

[Daniel (daniel-redhat-bugs)]

WRT to comment #5, if you re-configured libvirtd to run qemu as root:root, we now automatically set directory ownership correctly.

WRT to comment #2, I believe that one is the bug caused by accidentally using virtio for CDROMs also now fixed.

So at least the two workarounds ought to be operational now. Leave this one open to track improving the logic in libvirtd though as per comment #3

[Mark (mark-redhat-bugs)]

*** Bug 526199 has been marked as a duplicate of this bug. ***

[Mark (mark-redhat-bugs)]

*** Bug 527102 has been marked as a duplicate of this bug. ***

[Yanko (yanko-redhat-bugs)]

*** Bug 533649 has been marked as a duplicate of this bug. ***

[Mark (mark-redhat-bugs)]

*** Bug 538527 has been marked as a duplicate of this bug. ***

[Mark (mark-redhat-bugs)]

Note, the reporter of bug #538527 was having similar problems, but with using a samba mount

[Rubin (rubin-redhat-bugs)]

Hi, I'm the reporter of #538527 (my first bugreport using abrt - I had not realized the coolness of abrt until now ;-).

I've tested the behaviour on my samba share and on an nfs share. Behaviour is indeed the same. When I copy the images to my desktop system you can see that libvirt changes the ownership of the files:

[rubin@inden Desktop]$ ls -l
total 6584024
-rw-r--r-- 1 qemu qemu 3204427776 2009-11-19 11:29 Fedora-12-i386-DVD.iso
-rw-r--r-- 1 qemu qemu 3537600512 2009-11-19 11:37 Fedora-12-x86_64-DVD.iso

The owner was previously rubin:users. The mode was previously 664.

[Albert (albert-redhat-bugs-1)]

It's clear to me that libvirt should _not_ be changing the permissions of the source installation ISO image in the first place and this is the real bug.

By all means, change the ownership of the destination VM image (disk) being created, but not the source installation image or ISO.

A simply use case would be installing a VM from two or more different systems sharing an NFS based ISO image, in this case who owns the ISO image assuming the uid/gid differ on those systems.

The ISO image should have sufficient privileges to start with as set by the ADMIN otherwise contact your administrator (most likely yourself)

My guess is that the libvirt code just blindly sets the both the source and destination image permissions without much thought.

The Solution: Don't set the destination permissions from libvirt (or anything else)

and

if libvert cannot read the source image display a error message that humans can understand like "access denied" and not what it currently displays and I quote

"Unable to complete install: 'internal error unable to start guest: libvir: QEMU error : cannot set ownership on /datadisk/ISO/AsteriskNOW-1.5.0-i386-1of1.iso: Invalid argument"

The above message indicates another bug altogether, it implies I'm trying to start the image which I am not, I'm simply setting it up in virt-manager.

[Richard (richard-redhat-bugs)]

I have to agree with comment 13 that libvirt(d) really shouldn't be in the
business of chowning files. Even checking file permissions is dubious,
since to do a check you have to know everything about how file permissions,
SELinux, POSIX ACLs etc work.

[Daniel (daniel-redhat-bugs)]

*** Bug 552858 has been marked as a duplicate of this bug. ***

[Jens (jens-redhat-bugs)]

Any update?

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[John (john-redhat-bugs)]

I fully agree with comment #14 and comment #13. It's very common to have a local mirror of the ISOs shared via NFS (or Samba) so that repeated downloads aren't necessary for everyone at that site.

My work around for now is to make a copy on the local host in /tmp from the NFS share, which is just plain silly, sad, and embarrassing.

[Jens (jens-redhat-bugs)]

Any chance of a fix in the F13 timeframe? This is quite a pain point IMHO.

[Jens (jens-redhat-bugs)]

(Updating summary based on preceding comments.)

[Frank (frank-redhat-bugs)]

(In reply to comment #3)

> - Edit /etc/libvirt/qemu.conf and set it to run guest as user="root" and
> group="root", instead of qemu/qemu (ie reverting to < F11 behaviour)
>

This didn't work for me. same permission problem.
libvirt-0.7.7-5.fc13.x86_64
virt-manager-0.8.4-1.fc13.noarch

The nfs server is a NAS

[Gary (gary-redhat-bugs)]

Don't know if this is apparent to all people having this issue, but user qemu and group qemu MUST exist on the NFS server, or you can't set the permissions, even as root on a no_root_squash share. I don't know if that is the way NFS is SUPPOSE to work or not.

[Frank (frank-redhat-bugs)]

(In reply to comment #22)
> Don't know if this is apparent to all people having this issue, but user qemu
> and group qemu MUST exist on the NFS server, or you can't set the permissions,
> even as root on a no_root_squash share. I don't know if that is the way NFS is
> SUPPOSE to work or not.

Once I did that I was fine:
Using the nas (nfs server) web-interface created user=qemu groups=qemu\kvm
also created a secure folder for qemu.
Transferred all ~/Torrents to this new folder.
Adjusted virt-host /etc/fstab to suit.
virthost$ su
chown -R qemu:kvm /new_path/to/Torrents.

no permission problems since

[Marc (marc-redhat-bugs-1)]

Why should the iso need to have special permissions for qemu or kvm? Isn't it pretending the ISO is a CD? As long as it can read the ISO, there should be no need to change ownership of it.

[TakIchikawa (takichikawa-redhat-bugs)]

I was embarrassed, too.
However, I was able to 'not to change ownership' by editing the following parameter.

 vi /etc/libvirt/qemu.conf
    139 # Whether libvirt should dynamically change file ownership
    140 # to match the configured user/group above. Defaults to 1.
    141 # Set to 0 to disable file ownership changes.
    142 #dynamic_ownership = 1
    143 dynamic_ownership = 0

But, where this side effect occurs is not understood.
Is it good in this?

[Bug (bug-redhat-bugs)]

This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 13 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Jens (jens-redhat-bugs)]

Still happens with F15 as far as I can tell.

[Laine (laine-redhat-bugs)]

Please test this on F15 with libvirt-0.9.2 when it is available (the virt-preview repo for F15 is still at 0.9.1). The following commit may solve your problem:

commit 62ed801c13787cc844e75db7cd2d4c8a42454fcc
Author: Laine Stump <email address hidden>
Date: Fri Jun 3 11:59:09 2011 -0400

    security driver: ignore EINVAL when chowning an image file

    This fixes:

      https://bugzilla.redhat.com/show_bug.cgi?id=702044
      https://bugzilla.redhat.com/show_bug.cgi?id=709454

    Both of these complain of a failure to use an image file that resides
    on a read-only NFS volume. The function in the DAC security driver
    that chowns image files to the qemu user:group before using them
    already has special cases to ignore failure of chown on read-only file
    systems, and in a few other cases, but it hadn't been checking for
    EINVAL, which is what is returned if the qemu user doesn't even exist
    on the NFS server.

    Since the explanation of EINVAL in the chown man page almost exactly
    matches the log message already present for the case of EOPNOTSUPP,
    I've just added EINVAL to that same conditional.

As long as the qemu user/group has permission to read the ISO file with its existing user/group/mode on the server, any errors while attempting to chown/chgrp the file will be ignored, and libvirt will allow qemu to use the image (you don't even have to set dynamic_ownership=0).

If you're unfamiliar with the virt-preview repo, please check here:

   http://fedoraproject.org/wiki/Virtualization_Preview_Repository

[Fedora (fedora-redhat-bugs)]

This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.

[Laine (laine-redhat-bugs)]

This bug was actually fixed and verified quite a long time ago.

[Norbert (nrbrtx)]

Same with ISO placed on NTFS disk.
I use Ubuntu 12.04.5 LTS amd64 with virt-manager 0.9.1-1ubuntu5.1.

[Norbert (nrbrtx)]

The error was:

Unable to complete install: 'internal error Process exited while reading console log output: Error opening file for reading: Permission denied
char device redirected to /dev/pts/2
kvm-spice: -drive file=/media/WD2TB_NTFS/Distr/Linux/Debian+Ubuntu/ubuntu-14.04.4-desktop-i386.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw: could not open disk image /media/WD2TB_NTFS/Distr/Linux/Debian+Ubuntu/ubuntu-14.04.4-desktop-i386.iso: Permission denied
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1909, in do_install
    guest.start_install(False, meter=meter)
  File "/usr/lib/python2.7/dist-packages/virtinst/Guest.py", line 1236, in start_install
    noboot)
  File "/usr/lib/python2.7/dist-packages/virtinst/Guest.py", line 1304, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2166, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error Process exited while reading console log output: Error opening file for reading: Permission denied
char device redirected to /dev/pts/2
kvm-spice: -drive file=/media/WD2TB_NTFS/Distr/Linux/Debian+Ubuntu/ubuntu-14.04.4-desktop-i386.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw: could not open disk image /media/WD2TB_NTFS/Distr/Linux/Debian+Ubuntu/ubuntu-14.04.4-desktop-i386.iso: Permission denied