Curtin fails to deploy through MAAS due to tar error

[Lee Trager]

I just tried to deploy both trusty and wily using MAAS 1.9.0~alpha6+bzr4398 and curtin 0.1.0~bzr290 and both fail due to a tar error. Downgrading to curtin 0.1.0~bzr276(latest in maas-next-proposed) fixed the issue.

Related bugs:
 * bug 1511104: support more compression [xz or bzip] in curtin 'extract'

Get:1 http://security.ubuntu.com wily-security InRelease [64.4 kB]
Get:2 http://archive.ubuntu.com wily InRelease [218 kB]
Get:3 http://archive.ubuntu.com wily-updates InRelease [64.4 kB]
Get:4 http://security.ubuntu.com wily-security/main amd64 Packages [7161 B]
Get:5 http://security.ubuntu.com wily-security/universe amd64 Packages [2840 B]
Get:6 http://archive.ubuntu.com wily/main amd64 Packages [1420 kB]
Get:7 http://archive.ubuntu.com wily/universe amd64 Packages [6704 kB]
Get:8 http://archive.ubuntu.com wily-updates/main amd64 Packages [9190 B]
Get:9 http://archive.ubuntu.com wily-updates/universe amd64 Packages [3096 B]
Fetched 8492 kB in 2s (2986 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  dmeventd libdevmapper-event1.02.1 liblvm2cmd2.02 liblzo2-2 libreadline5
  module-init-tools postfix ssl-cert watershed
Suggested packages:
  thin-provisioning-tools procmail postfix-mysql postfix-pgsql postfix-ldap
  postfix-pcre sasl2-bin dovecot-common postfix-cdb mail-reader postfix-doc
  openssl-blacklist xfsdump attr quota
Recommended packages:
  default-mta mail-transport-agent
The following NEW packages will be installed:
  btrfs-tools dmeventd libdevmapper-event1.02.1 liblvm2cmd2.02 liblzo2-2
  libreadline5 lvm2 mdadm module-init-tools postfix ssl-cert watershed
  xfsprogs
0 upgraded, 13 newly installed, 0 to remove and 3 not upgraded.
Need to get 3956 kB of archives.
After this operation, 16.1 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ wily/main liblzo2-2 amd64 2.08-1.2 [48.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ wily/main module-init-tools all 21-1ubuntu1 [1946 B]
Get:3 http://archive.ubuntu.com/ubuntu/ wily/main btrfs-tools amd64 4.0-2 [475 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ wily/main libdevmapper-event1.02.1 amd64 2:1.02.99-1ubuntu1 [11.6 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ wily/main liblvm2cmd2.02 amd64 2.02.122-1ubuntu1 [495 kB]
Get:6 http://archive.ubuntu.com/ubuntu/ wily/main dmeventd amd64 2:1.02.99-1ubuntu1 [26.2 kB]
Get:7 http://archive.ubuntu.com/ubuntu/ wily/main libreadline5 amd64 5.2+dfsg-3 [101 kB]
Get:8 http://archive.ubuntu.com/ubuntu/ wily/main watershed amd64 7 [11.4 kB]
Get:9 http://archive.ubuntu.com/ubuntu/ wily/main lvm2 amd64 2.02.122-1ubuntu1 [702 kB]
Get:10 http://archive.ubuntu.com/ubuntu/ wily/main mdadm amd64 3.3-2ubuntu2 [395 kB]
Get:11 http://archive.ubuntu.com/ubuntu/ wily/main ssl-cert all 1.0.37 [16.9 kB]
Get:12 http://archive.ubuntu.com/ubuntu/ wily/main postfix amd64 2.11.3-1ubuntu2 [1097 kB]
Get:13 http://archive.ubuntu.com/ubuntu/ wily/main xfsprogs amd64 3.2.1ubuntu1 [575 kB]
Preconfiguring packages ...
Fetched 3956 kB in 3s (1223 kB/s)
Selecting previously unselected package liblzo2-2:amd64.

(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 61866 files and directories currently installed.)

Preparing to unpack .../liblzo2-2_2.08-1.2_amd64.deb ...

Unpacking liblzo2-2:amd64 (2.08-1.2) ...

Selecting previously unselected package module-init-tools.

Preparing to unpack .../module-init-tools_21-1ubuntu1_all.deb ...

Unpacking module-init-tools (21-1ubuntu1) ...

Selecting previously unselected package btrfs-tools.

Preparing to unpack .../btrfs-tools_4.0-2_amd64.deb ...

Unpacking btrfs-tools (4.0-2) ...

Selecting previously unselected package libdevmapper-event1.02.1:amd64.

Preparing to unpack .../libdevmapper-event1.02.1_2%3a1.02.99-1ubuntu1_amd64.deb ...

Unpacking libdevmapper-event1.02.1:amd64 (2:1.02.99-1ubuntu1) ...

Selecting previously unselected package liblvm2cmd2.02:amd64.

Preparing to unpack .../liblvm2cmd2.02_2.02.122-1ubuntu1_amd64.deb ...

Unpacking liblvm2cmd2.02:amd64 (2.02.122-1ubuntu1) ...

Selecting previously unselected package dmeventd.

Preparing to unpack .../dmeventd_2%3a1.02.99-1ubuntu1_amd64.deb ...

Unpacking dmeventd (2:1.02.99-1ubuntu1) ...

Selecting previously unselected package libreadline5:amd64.

Preparing to unpack .../libreadline5_5.2+dfsg-3_amd64.deb ...

Unpacking libreadline5:amd64 (5.2+dfsg-3) ...

Selecting previously unselected package watershed.

Preparing to unpack .../archives/watershed_7_amd64.deb ...

Unpacking watershed (7) ...

Selecting previously unselected package lvm2.

Preparing to unpack .../lvm2_2.02.122-1ubuntu1_amd64.deb ...

Unpacking lvm2 (2.02.122-1ubuntu1) ...

Selecting previously unselected package mdadm.

Preparing to unpack .../mdadm_3.3-2ubuntu2_amd64.deb ...

Unpacking mdadm (3.3-2ubuntu2) ...

Selecting previously unselected package ssl-cert.

Preparing to unpack .../ssl-cert_1.0.37_all.deb ...

Unpacking ssl-cert (1.0.37) ...

Selecting previously unselected package postfix.

Preparing to unpack .../postfix_2.11.3-1ubuntu2_amd64.deb ...

Unpacking postfix (2.11.3-1ubuntu2) ...

Selecting previously unselected package xfsprogs.

Preparing to unpack .../xfsprogs_3.2.1ubuntu1_amd64.deb ...

Unpacking xfsprogs (3.2.1ubuntu1) ...

Processing triggers for man-db (2.7.4-1) ...

Processing triggers for ureadahead (0.100.0-19) ...

Processing triggers for systemd (225-1ubuntu9) ...

Processing triggers for ufw (0.34-2) ...

Setting up liblzo2-2:amd64 (2.08-1.2) ...

Setting up module-init-tools (21-1ubuntu1) ...

Setting up btrfs-tools (4.0-2) ...

update-initramfs: deferring update (trigger activated)

Setting up libdevmapper-event1.02.1:amd64 (2:1.02.99-1ubuntu1) ...

Setting up libreadline5:amd64 (5.2+dfsg-3) ...

Setting up watershed (7) ...

update-initramfs: deferring update (trigger activated)

Setting up mdadm (3.3-2ubuntu2) ...

Generating mdadm.conf... done.

update-initramfs: deferring update (trigger activated)

/usr/sbin/grub-probe: error: failed to get canonical path of `overlayroot'.

invoke-rc.d: policy-rc.d denied execution of start.

update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults

Setting up ssl-cert (1.0.37) ...

hostname: Name or service not known

make-ssl-cert: Could not get FQDN, using "maas-test-1".

make-ssl-cert: You may want to fix your /etc/hosts and/or DNS setup and run

make-ssl-cert: make-ssl-cert generate-default-snakeoil --force-overwrite

make-ssl-cert: again.

Setting up postfix (2.11.3-1ubuntu2) ...

Adding group `postfix' (GID 118) ...

Done.

Adding system user `postfix' (UID 112) ...

Adding new user `postfix' (UID 112) with group `postfix' ...

ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Not creating home directory `/var/spool/postfix'.

Creating /etc/postfix/dynamicmaps.cf

Adding tcp map entry to /etc/postfix/dynamicmaps.cf

Adding sqlite map entry to /etc/postfix/dynamicmaps.cf

Adding group `postdrop' (GID 119) ...

Done.

setting myhostname: maas-test-1

setting alias maps

setting alias database

mailname is not a fully qualified domain name. Not changing /etc/mailname.

setting destinations: localdomain, localhost, localhost.localdomain, localhost

setting relayhost:

setting mynetworks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

setting mailbox_size_limit: 0

setting recipient_delimiter: +

setting inet_interfaces: all

setting inet_protocols: all

/etc/aliases does not exist, creating it.

WARNING: /etc/aliases exists, but does not have a root alias.

Postfix is now set up with a default configuration. If you need to make

changes, edit

/etc/postfix/main.cf (and others) as needed. To view Postfix configuration

values, see postconf(1).

After modifying main.cf, be sure to run '/etc/init.d/postfix reload'.

Running newaliases

invoke-rc.d: policy-rc.d denied execution of restart.

Setting up xfsprogs (3.2.1ubuntu1) ...

Setting up dmeventd (2:1.02.99-1ubuntu1) ...

/usr/sbin/policy-rc.d returned 101, not running 'start dm-event.socket'

Setting up lvm2 (2.02.122-1ubuntu1) ...

update-initramfs: deferring update (trigger activated)

invoke-rc.d: policy-rc.d denied execution of start.

invoke-rc.d: policy-rc.d denied execution of start.

Setting up liblvm2cmd2.02:amd64 (2.02.122-1ubuntu1) ...

Processing triggers for libc-bin (2.21-0ubuntu4) ...

Processing triggers for initramfs-tools (0.120ubuntu6) ...

update-initramfs: Generating /boot/initrd.img-4.2.0-16-generic

cryptsetup: WARNING: failed to detect canonical device of /media/root-ro/

cryptsetup: WARNING: could not determine root device from /etc/fstab

Warning: /sbin/fsck.overlayfs doesn't exist, can't install to initramfs, ignoring.

W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.

Processing triggers for ureadahead (0.100.0-19) ...

Processing triggers for systemd (225-1ubuntu9) ...

Processing triggers for ufw (0.34-2) ...

mdadm: No arrays found in config file or automatically
  Logical volume "lvroot" successfully removed
  Volume group "lvroot" not found
  Cannot process volume group lvroot
  Volume group "vgroot" successfully removed
  Physical volume "/dev/vda1" successfully created
  Volume group "vgroot" successfully created
  Logical volume "lvroot" created.
--2015-10-27 21:23:42-- http://10.0.0.2:5248/images/ubuntu/amd64/generic/wily/daily/root-tgz
Connecting to 10.0.0.2:5248... connected.
HTTP request sent, awaiting response... 200 OK
Length: 389807502 (372M) [text/html]
Saving to: 'STDOUT'
tar: Archive is compressed. Use -z option
tar: Error is not recoverable: exiting now

     0KUnexpected error while running command.
Command: ['sh', '-cf', 'wget "$1" --progress=dot:mega -O - |tar -C "$2" --xattrs --xattrs-include=* -Sxpf - --numeric-owner', '--', 'http://10.0.0.2:5248/images/ubuntu/amd64/generic/wily/daily/root-tgz', '/tmp/tmpo9rgo0es/target']
Exit code: 2
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'extract']
Exit code: 3
Reason: -
Stdout: b'--2015-10-27 21:23:42-- http://10.0.0.2:5248/images/ubuntu/amd64/generic/wily/daily/root-tgz\nConnecting to 10.0.0.2:5248... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 389807502 (372M) [text/html]\nSaving to: \'STDOUT\'\ntar: Archive is compressed. Use -z option\ntar: Error is not recoverable: exiting now\n\n 0KUnexpected error while running command.\nCommand: [\'sh\', \'-cf\', \'wget "$1" --progress=dot:mega -O - |tar -C "$2" --xattrs --xattrs-include=* -Sxpf - --numeric-owner\', \'--\', \'http://10.0.0.2:5248/images/ubuntu/amd64/generic/wily/daily/root-tgz\', \'/tmp/tmpo9rgo0es/target\']\nExit code: 2\nReason: -\nStdout: \'\'\nStderr: \'\'\n'
Stderr: ''

[Ryan Harper (raharper)]

The source of the change is here, the question is what isn't tar doing the right thing then.
My understanding is that tar can skip the compression option if it's operating on a file directly (tar -tvf mytar.tgz) but when passed through a pipe, it doesnt (cat mytar.tgz | tar -tvf -).

% bzr log -n0 -r282
------------------------------------------------------------
revno: 282
committer: Scott Moser <email address hidden>
branch nick: trunk
timestamp: Fri 2015-10-09 10:48:17 -0400
message:
  extract: do not specify 'z' to tar extract

  tar will automatically do the right thing for compression
  it knows about. Removing the 'z' here makes us able to install
  a .tar.xz file too.

% bzr diff -r281..282
=== modified file 'curtin/commands/extract.py'
--- curtin/commands/extract.py 2015-10-02 18:37:42 +0000
+++ curtin/commands/extract.py 2015-10-09 14:48:17 +0000
@@ -55,7 +55,7 @@
     curtin.util.subp(args=['sh', '-cf',
                            ('wget "$1" --progress=dot:mega -O - |'
                             'tar -C "$2" ' + ' '.join(tar_xattr_opts()) +
- ' ' + '-Sxpzf - --numeric-owner'),
+ ' ' + '-Sxpf - --numeric-owner'),
                            '--', source, target])

[Scott Moser (smoser)]

I sent mail to http://lists.gnu.org/archive/html/bug-tar/2015-10/index.html suggesting they implement essentially what smtar does. But thats a future thing very much.

[Scott Moser (smoser)]

I reverted revno 282 in revno 291.
So trunk does not suffer this bug any more.
I'll mark fixed, and open another for the 'smtar'.