Backport "libdpkg: Only preallocate disk size for files bigger than 16 KiB".

[Vinson Lee]

Please backport upstream dpkg patch "libdpkg: Only preallocate disk size for files bigger than 16 KiB" to xenial, yakkety, and zesty versions of dpkg.

dpkg 1.17.11 commit "dpkg: Try to preallocate the disk size for extracted files" causes installs of packages with many small files to be slow. https://anonscm.debian.org/git/dpkg/dpkg.git/commit/?id=87b0b20b86407baf1deb4e91b3fd839e01228ac8

This was fixed in dpkg 1.18.15 with "libdpkg: Only preallocate disk size for files bigger than 16 KiB". https://anonscm.debian.org/git/dpkg/dpkg.git/commit/?id=a971ad91437af8880cad4703695dcf12ee45959b

trusty dpkg is based off 1.17.5 and is not affected. xenial dpkg based off 1.18.4, yakkety dpkg based off 1.18.10, and zesty dpkg based off 1.18.10 need the fix.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Vinson Lee (vlee)]

Here is a test case than be used to benchmark the performance with and without the fix.

$ wget https://packages.chef.io/files/stable/chefdk/1.1.16/ubuntu/16.04/chefdk_1.1.16-1_amd64.deb
$ time sudo dpkg --install chefdk_1.1.16-1_amd64.deb

[Luke W Faraone (lwf)]

Triaging as high, because this has a moderate negative performance impact/regression on *all* Ubuntu users, and is adversely impacting at least one substantially large deployment's Xenial rollout. (albeit with a hacky workaround)

[Łukasz Zemczak (sil2100)]

Hello Vinson, or anyone else affected,

Accepted dpkg into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.18.4ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Vinson, or anyone else affected,

Accepted dpkg into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.18.10ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.18.10ubuntu2

---------------
dpkg (1.18.10ubuntu2) zesty; urgency=medium

  * Cherry-pick patch from git to limit preallocation to files over
    16KiB, fixing installation performance regression (LP: #1663749)

 -- Adam Conrad <email address hidden> Mon, 10 Apr 2017 04:08:19 -0600

[Łukasz Zemczak (sil2100)]

I was asked to test this bug on a slower non-SSD disk. Checked zesty, yakkety and xenial. The detailed test result:

zesty:
- 1.18.10ubuntu1:
real 0m39.559s
user 0m7.176s
sys 0m8.780s
- 1.18.10ubuntu2:
real 0m20.273s
user 0m6.096s
sys 0m7.164s

yakkety:
- 1.18.9ubuntu2
real 0m38.154s
user 0m7.008s
sys 0m9.112s
- 1.18.10ubuntu1.1
real 0m24.107s
user 0m6.028s
sys 0m7.116s

xenial:
- 1.18.4ubuntu1
real 0m41.030s
user 0m8.164s
sys 0m8.632s
- 1.18.4ubuntu1.2
real 0m19.263s
user 0m6.792s
sys 0m5.644s

Retried a few times afterwards to make sure the results are reproducible. With this, I consider the bug as verified for both stable releases.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.18.4ubuntu1.2

---------------
dpkg (1.18.4ubuntu1.2) xenial; urgency=medium

  * Cherry-pick patch from git to limit preallocation to files over
    16KiB, fixing installation performance regression (LP: #1663749)

 -- Adam Conrad <email address hidden> Mon, 10 Apr 2017 04:08:19 -0600

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for dpkg has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.18.10ubuntu1.1

---------------
dpkg (1.18.10ubuntu1.1) yakkety; urgency=medium

  * Cherry-pick patch from git to limit preallocation to files over
    16KiB, fixing installation performance regression (LP: #1663749)

 -- Adam Conrad <email address hidden> Mon, 10 Apr 2017 04:08:19 -0600