"Invalid cross-device link" and failed to preserve ownership error on image-output...

Bug #1637554 reported by david butler on 2016-10-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Image
Medium
Barry Warsaw

Bug Description

Here is your wall of traceback:

cro@ubuntu:~$ sudo /snap/bin/ubuntu-image \
> --extra-snaps=/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/linuxium-kernel_4.4.0-45.66_amd64.snap \
> -c beta -o ${DIR2_}/ics-ubuntu-image.img \
> ${DIR4_}/ics.model
Fetching core
Copying "/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/linuxium-kernel_4.4.0-45.66_amd64.snap" (linuxium-kernel)
Fetching pc
linuxium-kernel already prepared, skipping
COMMAND FAILED: ['cp', '-p', '--sparse=always', '/tmp/tmpsjo_rg8t/.images/disk.img', '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img']
cp: failed to preserve ownership for '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img': Operation not permitted
uncaught exception in state machine step: [11] finish
Traceback (most recent call last):
  File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpsjo_rg8t/.images/disk.img' -> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/state.py", line 68, in __next__
    step()
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/builder.py", line 430, in finish
    shutil.move(self.disk_img, self.output, copy_function=sparse_copy)
  File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 552, in move
    copy_function(src, real_dst)
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py", line 142, in sparse_copy
    run(args)
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py", line 120, in run
    proc.check_returncode()
  File "/snap/ubuntu-image/25/usr/lib/python3.5/subprocess.py", line 659, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['cp', '-p', '--sparse=always', '/tmp/tmpsjo_rg8t/.images/disk.img', '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img']' returned non-zero exit status 1
Crash in state machine
Traceback (most recent call last):
  File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpsjo_rg8t/.images/disk.img' -> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/__main__.py", line 171, in main
    list(state_machine)
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/state.py", line 68, in __next__
    step()
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/builder.py", line 430, in finish
    shutil.move(self.disk_img, self.output, copy_function=sparse_copy)
  File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 552, in move
    copy_function(src, real_dst)
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py", line 142, in sparse_copy
    run(args)
  File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py", line 120, in run
    proc.check_returncode()
  File "/snap/ubuntu-image/25/usr/lib/python3.5/subprocess.py", line 659, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['cp', '-p', '--sparse=always', '/tmp/tmpsjo_rg8t/.images/disk.img', '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img']' returned non-zero exit status 1

This fix worked for me:

cro@ubuntu:~$ diff /tmp/helpers.py /snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py
139c139
< args = ['cp', '--sparse=always', src, dst]
---
> args = ['cp', '-p', '--sparse=always', src, dst]

Steve Langasek (vorlon) wrote :

Hi David,

Thanks for the bug report. So, 'cp -p' should only fail if the target filesystem is not a POSIX filesystem. Is that the case for you? I see /home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img as a destination, I'm not sure if 'hgfs' refers to a particular filesystem with non-POSIX semantics?

This is certainly a legitimate bug. We may be able to drop the '-p' with the reasoning that a file copy will create the new file with the same umask and owner as the original file creation, but we need to check this assumption.

Changed in ubuntu-image:
status: New → Triaged
importance: Undecided → Medium
Download full text (5.7 KiB)

Hgfs is a file system that VMware fusion uses to mount a host file system
from OS X onto the guest...

On Friday, October 28, 2016, Steve Langasek <email address hidden>
wrote:

> Hi David,
>
> Thanks for the bug report. So, 'cp -p' should only fail if the target
> filesystem is not a POSIX filesystem. Is that the case for you? I see
> /home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/
> Signage/flock_installer
> /ics-ubuntu-image.img as a destination, I'm not sure if 'hgfs' refers to
> a particular filesystem with non-POSIX semantics?
>
> This is certainly a legitimate bug. We may be able to drop the '-p'
> with the reasoning that a file copy will create the new file with the
> same umask and owner as the original file creation, but we need to check
> this assumption.
>
> ** Changed in: ubuntu-image
> Status: New => Triaged
>
> ** Changed in: ubuntu-image
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1637554
>
> Title:
> "Invalid cross-device link" and failed to preserve ownership error on
> image-output...
>
> Status in Ubuntu Image:
> Triaged
>
> Bug description:
> Here is your wall of traceback:
>
> cro@ubuntu:~$ sudo /snap/bin/ubuntu-image \
> > --extra-snaps=/home/cro/home_root/mnt/2/snapcraft_kernel_
> build_test/linuxium-kernel_4.4.0-45.66_amd64.snap \
> > -c beta -o ${DIR2_}/ics-ubuntu-image.img \
> > ${DIR4_}/ics.model
> Fetching core
> Copying "/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/
> linuxium-kernel_4.4.0-45.66_amd64.snap" (linuxium-kernel)
> Fetching pc
> linuxium-kernel already prepared, skipping
> COMMAND FAILED: ['cp', '-p', '--sparse=always',
> '/tmp/tmpsjo_rg8t/.images/disk.img', '/home/cro/home_root/mnt/hgfs/
> cro/Documents/TanagerProjects/Signage/flock_installer/ics-
> ubuntu-image.img']
> cp: failed to preserve ownership for '/home/cro/home_root/mnt/hgfs/
> cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img':
> Operation not permitted
> uncaught exception in state machine step: [11] finish
> Traceback (most recent call last):
> File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 538,
> in move
> os.rename(src, real_dst)
> OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpsjo_rg8t/.images/disk.img'
> -> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/
> Signage/flock_installer/ics-ubuntu-image.img'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/state.py",
> line 68, in __next__
> step()
> File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/builder.py",
> line 430, in finish
> shutil.move(self.disk_img, self.output, copy_function=sparse_copy)
> File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 552,
> in move
> copy_function(src, real_dst)
> File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py",
> line 142, in sparse_copy
> ...

Read more...

Barry Warsaw (barry) on 2016-12-05
Changed in ubuntu-image:
milestone: none → 0.13
Barry Warsaw (barry) on 2016-12-13
Changed in ubuntu-image:
status: Triaged → In Progress
assignee: nobody → Barry Warsaw (barry)
Barry Warsaw (barry) wrote :

I wasn't able to get file sharing working with a Zesty guest in Fusion 8.5.3. Fusion kept complaining that it couldn't mount the host directory, so I suspect it's an incompatibility with open-vm-tools.

I *was* able to mount an afp: file system onto my Yakkety machine though and tries an equivalent command which moved the final disk image to the afp mounted directory. This reproduced the crash.

In this particular case, it looks like it's the copying of the timestamps that isn't supported. `cp -p` is equivalent to `cp --preserve=mode,ownership,timestamps` and if you substitute the former for the latter w/o timestamps (i.e. cp --preserve=mode,ownership) it works just fine. Preserving timestamps is effectively unnecessary since it's not terribly useful information and the cp is going to happen very soon after the original disk image creation anyway[*].

So I propose that simple change, although I won't be able to write a good test case for it.

[*] Unless you ran the state machine to the 'finish' step and then resumed it, but that's not a typical usage and again, I don't think preserving the timestamps is really all that useful.

Barry Warsaw (barry) wrote :

@croepha: Please test to see if --preserve-mode,ownership works for you.

Barry Warsaw (barry) wrote :

Actually, hmm. The cp error message is different in the afp case, so maybe hgfs has different semantics than afp after all.

Note that even with the operation not supported error, the resulting file is checksum equivalent, which means that `cp --preserve` is failing after the bits are copied. How much does it matter that the mode, ownership, and timestamp are preserved? I've already laid out why I think the timestamp doesn't matter as much. In my own tests over afp, the ownership of the new file is equivalent to the remote user's uid anyway. It's true that the remote file (after the exception) does not honor umask of either the source umask or target umask.

So another option would be: catch the CalledProcessError and if the checksums of the copied file are not equivalent to the source file, re-raise the exception. If they're equivalent, swallow the error.

Or we could just remove -p and be done with it.

david butler (croepha) wrote :
Download full text (6.8 KiB)

Barry:

I think it has specifically to do with root, and hgfs not allowing you to
set root ownership on hgfs. examine this:

cro@ubuntu:~$ touch test2
cro@ubuntu:~$ cp -p test2 /mnt/hgfs/cro/Documents/
cro@ubuntu:~$ sudo touch test3
cro@ubuntu:~$ sudo cp -p test3 /mnt/hgfs/cro/Documents/
cp: failed to preserve ownership for '/mnt/hgfs/cro/Documents/test3':
Operation not permitted
cro@ubuntu:~$ sudo touch test4
cro@ubuntu:~$ sudo cp --preserve=mode,ownership test4
/mnt/hgfs/cro/Documents/
cp: failed to preserve ownership for '/mnt/hgfs/cro/Documents/test4':
Operation not permitted
cro@ubuntu:~$

ask you can see, -p works fine for a normal user...

I would recommend just dropping it. I'm not sure what the added benefit of
'-p' is in a practical sense.... If the permissions in question were a
priority for the the user, than that user should select umasks and
filesystems appropriately.. I think the default functionality of `cp'
simply does the right thing...

On Tue, Dec 13, 2016 at 3:30 PM Barry Warsaw <email address hidden>
wrote:

> Actually, hmm. The cp error message is different in the afp case, so
> maybe hgfs has different semantics than afp after all.
>
> Note that even with the operation not supported error, the resulting
> file is checksum equivalent, which means that `cp --preserve` is failing
> after the bits are copied. How much does it matter that the mode,
> ownership, and timestamp are preserved? I've already laid out why I
> think the timestamp doesn't matter as much. In my own tests over afp,
> the ownership of the new file is equivalent to the remote user's uid
> anyway. It's true that the remote file (after the exception) does not
> honor umask of either the source umask or target umask.
>
> So another option would be: catch the CalledProcessError and if the
> checksums of the copied file are not equivalent to the source file, re-
> raise the exception. If they're equivalent, swallow the error.
>
> Or we could just remove -p and be done with it.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1637554
>
> Title:
> "Invalid cross-device link" and failed to preserve ownership error on
> image-output...
>
> Status in Ubuntu Image:
> In Progress
>
> Bug description:
> Here is your wall of traceback:
>
> cro@ubuntu:~$ sudo /snap/bin/ubuntu-image \
> >
> --extra-snaps=/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/linuxium-kernel_4.4.0-45.66_amd64.snap
> \
> > -c beta -o ${DIR2_}/ics-ubuntu-image.img \
> > ${DIR4_}/ics.model
> Fetching core
> Copying
> "/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/linuxium-kernel_4.4.0-45.66_amd64.snap"
> (linuxium-kernel)
> Fetching pc
> linuxium-kernel already prepared, skipping
> COMMAND FAILED: ['cp', '-p', '--sparse=always',
> '/tmp/tmpsjo_rg8t/.images/disk.img',
> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img']
> cp: failed to preserve ownership for
> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img':
> Operation not permitted
> unc...

Read more...

Barry Warsaw (barry) wrote :

David, thanks for the additional information.

On Dec 14, 2016, at 01:06 AM, david butler wrote:

>I would recommend just dropping it. I'm not sure what the added benefit of
>'-p' is in a practical sense.... If the permissions in question were a
>priority for the the user, than that user should select umasks and
>filesystems appropriately.. I think the default functionality of `cp'
>simply does the right thing...

We pretty much came to the same conclusion. :) We're dropping it.

Barry Warsaw (barry) on 2016-12-14
Changed in ubuntu-image:
status: In Progress → Fix Committed
Barry Warsaw (barry) on 2017-01-09
Changed in ubuntu-image:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers