Please use xz instead of pxz

Bug #1701132 reported by Balint Reczey on 2017-06-29
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
livecd-rootfs (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Balint Reczey

Bug Description

[Impact]

 * Livecd-rootfs switches from using pxz to xz
 * Pxz is installed at run-time instead of being a package dependency and pxz is in universe despite livecd-rootfs being in main. Both of those are undesired and fixing them makes livecd-rootfs easier to maintain following best practice and policy in Ubuntu.

[Test Case]

 * Livecd-rootfs in Xenial uses (p)xz only for compressing raspi2 images.
   To test the change install updated livecd-rootfs on armhf and create images setting the following environment variables: ARCH=armhf SUBARCH=raspi2 PROJECT=ubuntu-cpc.
   Notice that the xz-compressed rapi2 image is still created.
 * Test-build Xenial image and make sure they are not broken. A subset of those images are built in the autopkgtest suite.

[Regression Potential]

 * It is a known and accepted regression that raspi2 image generation takes more wall-clock time after switching to xz because while Xenial's xz version does accept the -T4 option it does not compress in a parallel manner.
 * The original change in Artful set -0 compression for xz which resulted in ~35% bigger compressed images. This regression is fixed in LP: #1732681 which is scheduled to be back-ported together with switching to xz.
 * In case of this change causes regression it would manifest itself as failing build for raspi2, failing build for ubuntu-cpc images or increased raspi2 image size.

[Original bug report text]

Livecd-rootfs (main) 2.447 uses "pxz -T4" for xz compression, while pxz is not in main.
Xz 5.2 also supports multi-threaded compression thus xz could easily replace current pxz usage.

A quick test on compressing livecd.ubuntu-cpc.img (created by ubuntu-image) shows that there is very little difference in compressed file size while xz's -T4 brings substantial speedup in wall clock time:

$ cat test-xz
#!/bin/sh

TESTFILE=$1
echo "Compr.\tThreads\tWall.t\tUser.t\tSys.t\tRes.t" > times.txt
echo "Wall.t\tUser.t\tSys.t\tRes.t" > decompression-times.txt
echo "Size(kB)" > sizes.txt
for threads in 1 4; do
    for compression in $(seq 0 9) $(seq -f '%0.fe' 0 9); do
        /usr/bin/time -a -o times.txt -f "-$compression\t -T$threads\t%e\t%U\t%S\t%M" -- xz -T$threads -$compression -k $TESTFILE
        /usr/bin/time -a -o decompression-times.txt -f "%e\t%U\t%S\t%M" -- xz -t $TESTFILE.xz
        ls -k -s $TESTFILE.xz | cut -d' ' -f1 >> sizes.txt
        rm $TESTFILE.xz
    done
done
paste times.txt decompression-times.txt sizes.txt > time-size.txt

$ cat time-size.txt
Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB)
-0 -T1 63.48 63.07 0.40 4932 1.25 1.16 0.08 2432 416324
-1 -T1 102.90 102.26 0.63 11064 1.53 1.26 0.14 3252 414624
-2 -T1 131.04 130.40 0.62 18544 1.43 1.27 0.09 4224 414616
-3 -T1 146.22 145.61 0.60 33992 1.43 1.24 0.12 6364 414616
-4 -T1 142.24 141.76 0.47 50488 1.34 1.21 0.10 6424 414584
-5 -T1 147.81 147.26 0.53 97616 1.37 1.22 0.10 10420 414564
-6 -T1 148.61 148.06 0.54 97608 1.35 1.23 0.10 10536 414564
-7 -T1 154.29 153.70 0.58 191800 1.44 1.22 0.14 18820 414564
-8 -T1 155.82 155.16 0.65 380336 1.53 1.27 0.13 35208 414564
-9 -T1 194.08 193.18 0.88 691128 1.59 1.31 0.13 67700 414444
-0e -T1 76.81 76.31 0.48 6296 1.27 1.21 0.05 2536 416256
-1e -T1 116.49 115.96 0.53 15068 1.28 1.20 0.07 3232 414568
-2e -T1 137.13 136.62 0.50 26892 1.30 1.20 0.10 4292 414564
-3e -T1 144.34 143.84 0.49 50556 1.31 1.21 0.09 6340 414564
-4e -T1 146.93 146.48 0.44 50412 1.30 1.20 0.10 6340 414568
-5e -T1 150.59 150.04 0.53 97552 1.32 1.22 0.10 10416 414564
-6e -T1 153.40 152.77 0.62 97564 1.32 1.22 0.09 10404 414568
-7e -T1 157.10 156.46 0.62 191956 1.35 1.26 0.08 18548 414568
-8e -T1 160.36 159.66 0.69 380248 1.46 1.25 0.13 35040 414568
-9e -T1 198.32 197.42 0.88 691256 1.49 1.19 0.14 67648 414448
-0 -T4 21.97 86.83 0.40 24896 1.28 1.22 0.05 2448 416368
-1 -T4 35.73 140.12 0.45 73456 1.29 1.22 0.06 3136 414640
-2 -T4 38.15 149.36 0.54 141044 1.30 1.21 0.08 4160 414624
-3 -T4 40.74 153.95 0.55 276336 1.30 1.22 0.08 6216 414620
-4 -T4 40.02 152.20 0.63 341980 1.38 1.23 0.11 6352 414588
-5 -T4 41.68 153.74 1.10 676656 1.49 1.27 0.10 10336 414564
-6 -T4 42.70 157.64 1.19 677144 1.48 1.25 0.13 10436 414564
-7 -T4 47.70 157.42 2.24 1330432 1.60 1.32 0.08 18480 414560
-8 -T4 74.56 178.48 13.32 1676392 1.69 1.34 0.10 35000 414560
Command terminated by signal 9 Command exited with non-zero status 1 0
-9 -T4 19.60 41.90 5.28 1656776 0.01 0.00 0.00 2076 416296
-0e -T4 30.23 117.70 0.56 30596 1.29 1.22 0.06 2444 414584
-1e -T4 37.16 144.83 0.51 90304 1.31 1.21 0.10 3272 414572
-2e -T4 38.73 151.84 0.50 174124 1.32 1.24 0.08 4296 414568
-3e -T4 40.78 156.10 0.65 342012 1.33 1.27 0.05 6432 414572
-4e -T4 42.03 161.22 0.62 342108 1.33 1.24 0.09 6396 414564
-5e -T4 43.10 160.05 1.08 676732 1.47 1.30 0.07 10448 414568
-6e -T4 43.17 159.25 1.31 676584 1.55 1.27 0.12 10328 414564
-7e -T4 48.12 162.80 2.09 1330552 1.60 1.29 0.12 18596 414568
-8e -T4 77.13 182.30 13.98 1673832 1.69 1.34 0.10 34920 0
Command terminated by signal 9 Command exited with non-zero status 1
-9e -T4 18.92 44.75 4.78 1662540 0.02 0.00 0.00 2196

Related branches

Balint Reczey (rbalint) wrote :
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.448

---------------
livecd-rootfs (2.448) artful; urgency=medium

  [ Balint Reczey ]
  * Depend on packages in main instead of installing them at runtime
  * Depend on vmdk-stream-converter instead of installing it at run time
  * Use xz -T4 -0 instead of pxz (LP: #1701132)
    Supported xz versions before 5.2 also accept the -T4 option but ignore
    it.
  * Depend on xz-utils instead of pxz

  [ Mathieu Trudel-Lapierre ]
  * auto/config: enable Debug::pkgDepCache::AutoInstall so tracking down
    dependency bugs in the livefs is a little clearer.

  [ Robert C Jennings ]
  * live-build/functions: Update path for VMDKstream.py (LP: #1706116)

 -- Brian Murray <email address hidden> Mon, 24 Jul 2017 11:06:24 -0700

Changed in livecd-rootfs (Ubuntu):
status: New → Fix Released
Balint Reczey (rbalint) wrote :

I tested the change with the image we actually ship with .xz compression and to my greatest surprise xz -0 actually performed much better than pxz with default compression:

$ du -sh *
1.2G ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img
243M ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.orig.xz
243M ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz
168M ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.xz

Balint Reczey (rbalint) wrote :

Please disregard my previous result, the xz -0 does generate a bigger .xz , it was a test running in parallel which updated the .xz file.

I'll attach the full test results when it is finished, but the actual file size was bigger with xz -0:
323M ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.xz

Balint Reczey (rbalint) wrote :
Download full text (3.3 KiB)

The original command's CPU usage is the following in xenial:
rbalint@xenial-test:~/Downloads$ time pxz -T4 -c ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img > ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz

real 3m31.117s
user 10m12.812s
sys 0m17.988s

rbalint@xenial-test:~/Downloads$ ls -al ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz
-rw-rw-r-- 1 rbalint rbalint 254621780 Nov 16 00:03 ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz

On the same system I retested the matrix of xz options and it shows that -T is ignored and -3 has the closest wall clock time with 258680 kB result which is slightly larger than the original but even -8 consumes system time and creates smaller compressed image:

rbalint@xenial-test:~/Downloads/xz-test$ cat time-size.txt
Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB)
-0 -T1 82.59 74.28 1.51 4792 25.38 25.16 0.10 2404 329984
-1 -T1 109.10 96.64 1.63 10712 23.63 22.55 0.08 3080 298056
-2 -T1 136.69 127.53 1.78 18392 22.36 21.71 0.08 4196 286276
-3 -T1 164.91 158.57 1.64 33948 21.46 20.93 0.09 6260 275036
-4 -T1 320.00 314.05 1.90 50320 21.18 20.86 0.13 6120 258680
-5 -T1 375.79 370.80 1.82 97480 20.46 20.23 0.10 10340 247316
-6 -T1 476.02 471.64 2.06 97416 20.45 20.18 0.10 10352 245768
-7 -T1 497.44 492.16 2.35 191624 21.24 20.06 0.09 18408 240188
-8 -T1 513.29 507.79 2.24 380108 19.43 19.16 0.10 34908 229736
-9 -T1 873.02 579.58 7.55 690952 22.61 19.30 0.14 67700 224492
-0e -T1 746.35 743.14 1.76 6224 23.69 23.32 0.11 2420 301004
-1e -T1 809.93 806.56 1.98 14988 21.57 21.31 0.11 3180 272976
-2e -T1 846.12 843.60 1.82 26748 22.10 20.75 0.07 4140 262860
-3e -T1 699.59 696.83 1.97 50316 20.90 20.68 0.07 6260 252052
-4e -T1 879.00 875.42 2.04 50256 20.35 19.96 0.10 6292 252048
-5e -T1 723.80 720.15 1.81 97480 21.35 20.37 0.11 10360 245172
-6e -T1 911.69 908.11 2.03 97464 20.37 19.83 0.07 10348 245168
-7e -T1 942.32 936.49 2.09 191616 19.72 19.25 0.12 18352 239552
-8e -T1 967.20 962.26 2.42 380156 18.85 18.07 0.14 34904 228956
-9e -T1 1177.95 1036.56 6.36 690784 19.78 18.08 0.11 67680 223748
-0 -T4 81.56 73.64 1.46 4820 29.27 25.82 0.09 2252 329984
-1 -T4 103.68 95.14 1.67 10800 24.41 22.86 0.12 3180 298060
-2 -T4 134.05 126.22 1.76 18480 22.06 21.65 0.13 4232 286276
-3 -T4 170.57 162.72 1.72 33812 21.62 20.94 0.12 6188 275036
-4 -T4 322.15 317.83 2.10 50284 21.55 21.05 0.07 6136 258684
-5 -T4 407.15 399.93 1.92 97656 21.85 20.50 0.12 10412 247320
-6 -T4 478.64 473.32 1.95 97660 21.92 20.34 0.09 10348 245772
-7 -T4 493.97 485.52 1.99 191692 20.66 20.01 0.08 18548 240192
-8 -T4 512.28 506.08 2.20 380096 22.03 19.58 0.11 34888 229740
-9 -T4 652.71 559.20 4.78 691076 20.57 19.15 0.08 67700 224496
-0e -T4 750.00 743.68 1.74 6172 24.59 23.34 0.11 2444 301004
-1e -T4 812.97 805.57 1.94 15004 21.81 21.36 0.11 3280 272976
-2e -T4 849.79 842.25 2.05 26828 20.96 20.63 0.12 4312 262860
-3e -T4 702.49 696.67 1.69 50380 20.85 20.61 0.09 6360 252056
-4e -T4 882.09 874.62 1.97 50552 20.11 19.89 0.08 6292 252052
-5e -T4 725.29 719.12 1.90 97512 20.42 20.20 0.06 10320 245176
-6e -T4 917.31 908.11 1.87 97632 20.26 19.78 0.12 10312 245...

Read more...

Balint Reczey (rbalint) wrote :

I opened LP: #1732681 to clarify if the regression is acceptable or xz should use a different compression than -0.

Balint Reczey (rbalint) on 2017-12-05
description: updated
Changed in livecd-rootfs (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Balint Reczey (rbalint)
Balint Reczey (rbalint) on 2017-12-05
description: updated
Balint Reczey (rbalint) on 2017-12-06
Changed in livecd-rootfs (Ubuntu Xenial):
status: In Progress → Fix Committed

Hello Balint, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed verification-needed-xenial
Chris Glass (tribaal) wrote :

Verified by building cloud images using the proposed package and ensuring the resulting artifacts were satisfactory.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Chris Glass (tribaal) wrote :

(verified with 2.408.25)

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.408.25

---------------
livecd-rootfs (2.408.25) xenial; urgency=medium

  * Sort and list dependencies nicely
  * Depend on packages instead of installing them at run time
  * Use xz -T4 instead of pxz.
    Supported xz versions before 5.2 accept the -T4 option but ignore it.
    Also depend on xz-utils instead of downloading pxz. (LP: #1701132)

 -- Balint Reczey <email address hidden> Wed, 06 Dec 2017 11:34:04 +0100

Changed in livecd-rootfs (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers