On Ubuntu 18.04 virt-copy-out fails with "tar: This does not look like a tar archive"

[Barry]

[Impact]

 * When building the virtual appliance for Ubuntu it fails to run
   almost any internal commands due to missing /bin/sh. Due to that any
   user facing action requireing those like the reported tar-copy will
   fail.

 * backport upstream fix that will make dash available

[Test Case]

 * Install target release (not well testable in VMs [needs working nesting] or Containers [mounts and virt permissions])

$ sudo apt install libguestfs-tools qemu-utils
$ qemu-img create -f qcow2 test.img 10M
$ sudo guestfish -a test.img
(guestfish) run
(guestfish) tar-out / - | tar tf -

[Regression Potential]

 * The change effectively "only" installs another package in the virtual
   appliance it uses to make up the environments. It currently is not
   working at all for anything subcommand that needs this as /bin/sh is
   missing.
   Due to that sub-functions of libguestfs-tools are either
    a) not using this part of it and not affected
    b) using the packagelist but broken as of today
   I can't see much of a regression risk as we only change those (fix
   actually) who are broken today.

[Other Info]

 * Cosmic had the change for quite a while as it was backported to 1.38.1
   and I heard no bad reports on it.

----

Version-Release number of selected component (if applicable):
Tested with:
- libguestfs 1.36.13 (Ubuntu 18.04)
- tar v1.29 (Ubuntu 18.04)

How reproducible: 100%

Steps to Reproduce:
1. # virt-builder debian-8 -o deb8.img --no-network
2. # mkdir ~/deb8fs
3. # virt-copy-out -a deb8.img / ~/deb8fs

Actual results:

libguestfs: error: file receive cancelled by daemon
libguestfs: error: tar subprocess failed: tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

See attached debug.txt log that was produced by using "export LIBGUESTFS_DEBUG=1 and LIBGUESTFS_TRACE=1" before running

**Note**

I have downloaded 1.39.9 from github/libguestfs and this version works without an error in my build directory.

It appears that the actual problem is not with "virt-copy-out" specifically, but with the 'guestfish' which it relies on.

Running this on 1.36.13 (Ubuntu) fails:

# guestfish
><fs> add-drive deb8.img
><fs> run
><fs> tar-out / - | tar tf -
libguestfs: error: file receive cancelled by daemon
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Whereas running the above on guestfish 1.39.9 from git does not produce that error.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libguestfs-tools 1:1.36.13-1ubuntu3.1
ProcVersionSignature: Ubuntu 4.15.0-34.37-generic 4.15.18
Uname: Linux 4.15.0-34-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 2.20.9-0ubuntu7.3
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Mon Sep 17 23:06:00 2018
InstallationDate: Installed on 2018-09-10 (7 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
ProcEnviron:
 LANG=en_US.UTF-8
 TERM=xterm-256color
 SHELL=/bin/bash
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, user)
SourcePackage: libguestfs
UpgradeStatus: No upgrade log present (probably fresh install)

[Richard Jones (rjones-redhat)]

I've been playing with this on an Ubuntu 18.04 machine this morning. Firstly I can confirm the bug with libguestfs0 1:1.36.13-1ubuntu3.1. Secondly I can confirm this does NOT affect libguestfs 1.39.9 built from source (from git).

[Richard Jones (rjones-redhat)]

I bisected this to:

commit 0cca7f9061455d29a7f893e715028cd3e75c19fd
Author: Pino Toscano <email address hidden>
Date: Mon Feb 19 14:46:32 2018 +0100

    appliance: include dash for Debian distros

(https://github.com/libguestfs/libguestfs/commit/0cca7f9061455d29a7f893e715028cd3e75c19fd)

This commit cherry picks cleanly on top of 1.36. I have verified that cherry picking this commit on top of 1.36 fixes the bug on Ubuntu 18.04.

[Richard Jones (rjones-redhat)]

I should add that there is a simple end user workaround:

echo dash > /usr/lib/guestfs/supermin.d/zz-lp-1793056

[Christian Ehrhardt  (paelzer)]

Hi,
could you check the version 1:1.38.4-1ubuntu1 that is in Ubuntu Bionic if it is affected as well?

For the fix we'd have to find the actual upstream change that fixed it to backport that, as the SRU policy [1] doesn't allow to push all of the new version.

With some luck later versions of the package are already fixed in Ubuntu.
That would be nice to know as we likely easier backport there and it makes the window of commits to search for smaller.

Note: This package is "only" in universe, so it depends on the community to fix it in cases like such. I'm willing to help as I can, but I can't promise anything.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Christian Ehrhardt  (paelzer)]

hah, due to being on business travel I needed so long to race your update :-)
Thanks a lot for the bisect, I'll take a look when I can to package that up.

[Barry (barry-magicrd)]

Richard:

Thanks for the help in figuring out this easy workaround.

Just a note, for me the solution was:

# echo dash > /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages

[Richard Jones (rjones-redhat)]

That probably deletes the original packages file, which isn't good. Would
definitely suggest creating a new file in the supermin.d directory.

[Barry (barry-magicrd)]

Sorry, had a typo in my post above...I appended (>>), I didn't overwrite (>).

However, point taken about creating a new file. I've done that and is working. Thanks.

For anyone coming from google, the temp solution on Ubuntu 18.04 is:

# echo dash > /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/zz-lp-1793056

(Just remember to clean up zz-lp-1793056 once the problem is fixed in future package release).

[Christian Ehrhardt  (paelzer)]

I took an older .img file I had available (any should do) and was able to confirm with just guestfish.

From there I tried to simplify the case and reached this:

$ qemu-img create -f qcow2 test.img 10M
$ sudo guestfish -a test.img
(guestfish) run
(guestfish) tar-out / - | tar tf -

I was able to confirm the case and the workaround helped.
I should soon be able to build a version which has the addition of dash in the package, but not sure if it will be today.

[Christian Ehrhardt  (paelzer)]

Confirmed to be an issue on Bionic

The upstream fix is in v1.39.1 originally, but was also picked for 1.38.1.
Due to that Cosmic already has the fix that is required.

Note: I verified that it works on cosmic and it does work

[Christian Ehrhardt  (paelzer)]

Test PPA at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3439
MP at: https://code.launchpad.net/~paelzer/ubuntu/+source/libguestfs/+git/libguestfs/+merge/355588

Waiting for MP review now, feel free to test from the PPA as well if time permits.

[Christian Ehrhardt  (paelzer)]

Uploaded, and waiting in Bionic-unapproved for the SRU team review

[Brian Murray (brian-murray)]

Hello Barry, or anyone else affected,

Accepted libguestfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libguestfs/1:1.36.13-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

Pre-Upgrade to proposed:

sudo guestfish -a test.img

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
><fs> tar-out / - | tar tf -
libguestfs: error: file receive cancelled by daemon
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
><fs>

---
Upgrade

$ sudo apt install libguestfs0 libguestfs-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libguestfs-hfsplus libguestfs-perl libguestfs-reiserfs libguestfs-xfs
Suggested packages:
  libguestfs-gfs2 libguestfs-jfs libguestfs-nilfs libguestfs-rescue libguestfs-rsync libguestfs-zfs
The following packages will be upgraded:
  libguestfs-hfsplus libguestfs-perl libguestfs-reiserfs libguestfs-tools libguestfs-xfs libguestfs0
6 upgraded, 0 newly installed, 0 to remove and 36 not upgraded.
Need to get 7727 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe ppc64el libguestfs-xfs ppc64el 1:1.36.13-1ubuntu3.2 [1784 B]
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe ppc64el libguestfs-tools ppc64el 1:1.36.13-1ubuntu3.2 [5089 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe ppc64el libguestfs-perl ppc64el 1:1.36.13-1ubuntu3.2 [301 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe ppc64el libguestfs-reiserfs ppc64el 1:1.36.13-1ubuntu3.2 [1784 B]
Get:5 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe ppc64el libguestfs-hfsplus ppc64el 1:1.36.13-1ubuntu3.2 [1780 B]
Get:6 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe ppc64el libguestfs0 ppc64el 1:1.36.13-1ubuntu3.2 [2332 kB]
Fetched 7727 kB in 3s (2396 kB/s)
(Reading database ... 73754 files and directories currently installed.)
Preparing to unpack .../0-libguestfs-xfs_1%3a1.36.13-1ubuntu3.2_ppc64el.deb ...
Unpacking libguestfs-xfs:ppc64el (1:1.36.13-1ubuntu3.2) over (1:1.36.13-1ubuntu3.1) ...
Preparing to unpack .../1-libguestfs-tools_1%3a1.36.13-1ubuntu3.2_ppc64el.deb ...
Unpacking libguestfs-tools (1:1.36.13-1ubuntu3.2) over (1:1.36.13-1ubuntu3.1) ...
Preparing to unpack .../2-libguestfs-perl_1%3a1.36.13-1ubuntu3.2_ppc64el.deb ...
Unpacking libguestfs-perl (1:1.36.13-1ubuntu3.2) over (1:1.36.13-1ubuntu3.1) ...
Preparing to unpack .../3-libguestfs-reiserfs_1%3a1.36.13-1ubuntu3.2_ppc64el.deb ...
Unpacking libguestfs-reiserfs:ppc64el (1:1.36.13-1ubuntu3.2) over (1:1.36.13-1ubuntu3.1) ...
Preparing to unpack .../4-libguestfs-hfsplus_1%3a1.36.13-1ubuntu3.2_ppc64el.deb ...
Unpacking libguestfs-hfsplus:ppc64el (1:1.36.13-1ubuntu3.2) over (1:1.36.13-1ubuntu3.1) ...
Preparing to unpack .../5-libguestfs0_1%3a1.36.13-1ubuntu3.2_ppc64el.deb ...
Unpacking libguestfs0:ppc64el...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libguestfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libguestfs - 1:1.36.13-1ubuntu3.2

---------------
libguestfs (1:1.36.13-1ubuntu3.2) bionic; urgency=medium

  * d/p/lp-1793056-fix-internal-bin-sh.patch: fix guestfish tasks that
    required virtual environment and missed /bin/sh (LP: #1793056)

 -- Christian Ehrhardt <email address hidden> Tue, 25 Sep 2018 09:52:15 +0200