Never-MarkAuto-Sections not working correctly

[Adam Conrad]

[ SRU Justification ]
Not only is this behaviour broken for every day use, but it also completely breaks livefs generation for ubuntukylin, and causes most of $flavour-desktop to be on the autoremove list in any point release livefs, making this bug point-release critical.

[ Test Case ]
Install a metapackage (say, ubuntu-standard), and then "apt-get --purge autoremove ubuntu-standard" and check that only that package is removed, rather than large chunks of its dependencies.

Further, also run the test case from comment 2 of bug #1429041 and make sure the previous fix didn't regress.

[ Regression Potential ]
My fix was tested locally against both test cases and passes with flying colours, so I don't see how this can make things any worse than they already are.

[ Original Report ]
In current trusty, vivid, and wily, when you install ubuntu-minimal (from the "metapackages" section) in a clean chroot, some 75ish packages are installed. If you then "apt-get --purge autoremove ubuntu-minimal", some large portion (between 20 and 40, depending on the target release) of those dependencies are removed again.

If this test is repeated putting the release pocket version of apt/libapt on hold, the purge/autoremove only removes the metapackage, but none of its dependencies, so this is likely a regression brought on from the previous autoremoval fix.

This bug is release-critical for the 14.04.3 point release, as we install using metapackages instead of tasks in point releases, thus meaning that anyone removing ubuntu-desktop for any reason will end up with dozens (potentially hundreds) of packages on the autoremoval list.

[David Kalnischkies (donkult)]

Without verifying if these Ubuntu versions actually have my bad patch ("deprecate the Section member from package struct" - fixing the problem of a package changing sections basically being randomly in one of these sections) I would presume this is debbug #793360 aka: Never-Mark-Auto doesn't apply on the dependencies of packages with the given sections, but on the packages with the given section itself. Much more details there.

[Adam Conrad (adconrad)]

Ah-ha, thanks for that pointer, David. We are, indeed, carrying that broken patch, and I can infer from Raphaël's overreaching patch (and followup discussion) what you're more likely to do as a conservative stable update. So, if you don't get around to said stable update, I'll cook up some SRUs for Ubuntu and see how they go.

[Adam Conrad (adconrad)]

Okay, based on the conversation in the Debian bug, I have a patch that fixes this regression, and doesn't regress the previous fix. SRUing to trusty to fix my point release, and will cover the other releases and maybe even write test cases for the testsuite when things calm down a tiny bit.

[Martin Pitt (pitti)]

Hello Adam, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

Next three comments are regression testing that the fix from #1429041 hasn't regressed.

[Adam Conrad (adconrad)]

(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# dpkg -l apt apt-utils libapt-inst1.5 libapt-pkg4.12 | awk '/^.i/ {print $2 " " $3}'
apt 1.0.1ubuntu2
apt-utils 1.0.1ubuntu2
libapt-inst1.5:amd64 1.0.1ubuntu2
libapt-pkg4.12:amd64 1.0.1ubuntu2
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# echo "deb file:///home/adconrad/apt-test/ ./" > /etc/apt/sources.list.d/test.list
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# cp Packages-broken Packages
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get update
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install test-meta
The following NEW packages will be installed:
  test-kernel test-meta
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-meta
The following packages will be REMOVED:
  test-meta*
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-kernel
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# cp Packages-notbroken Packages
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get update
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install test-meta
The following NEW packages will be installed:
  test-kernel test-meta
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-kernel
The following packages will be REMOVED:
  test-kernel* test-meta*
[...]

[Adam Conrad (adconrad)]

(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# dpkg -l apt apt-utils libapt-inst1.5 libapt-pkg4.12 | awk '/^.i/ {print $2 " " $3}'
apt 1.0.1ubuntu2.8
apt-utils 1.0.1ubuntu2.8
libapt-inst1.5:amd64 1.0.1ubuntu2.8
libapt-pkg4.12:amd64 1.0.1ubuntu2.8
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# echo "deb file:///home/adconrad/apt-test/ ./" > /etc/apt/sources.list.d/test.list
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# cp Packages-broken Packages
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get update
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install test-meta
The following NEW packages will be installed:
  test-kernel test-meta
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-meta
The following packages will be REMOVED:
  test-kernel* test-meta*
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# cp Packages-notbroken Packages
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get update
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install test-meta
The following NEW packages will be installed:
  test-kernel test-meta
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-kernel
The following packages will be REMOVED:
  test-kernel* test-meta*
[...]

[Adam Conrad (adconrad)]

(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# dpkg -l apt apt-utils libapt-inst1.5 libapt-pkg4.12 | awk '/^.i/ {print $2 " " $3}'
apt 1.0.1ubuntu2.9
apt-utils 1.0.1ubuntu2.9
libapt-inst1.5:amd64 1.0.1ubuntu2.9
libapt-pkg4.12:amd64 1.0.1ubuntu2.9
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# echo "deb file:///home/adconrad/apt-test/ ./" > /etc/apt/sources.list.d/test.list
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# cp Packages-broken Packages
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get update
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install test-meta
The following NEW packages will be installed:
  test-kernel test-meta
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-meta
The following packages will be REMOVED:
  test-kernel* test-meta*
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# cp Packages-notbroken Packages
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get update
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install test-meta
The following NEW packages will be installed:
  test-kernel test-meta
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove test-kernel
The following packages will be REMOVED:
  test-kernel* test-meta*
[...]

[Adam Conrad (adconrad)]

And the next three comments are regression testing for this current bug:

[Adam Conrad (adconrad)]

(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# dpkg -l apt apt-utils libapt-inst1.5 libapt-pkg4.12 | awk '/^.i/ {print $2 " " $3}'
apt 1.0.1ubuntu2
apt-utils 1.0.1ubuntu2
libapt-inst1.5:amd64 1.0.1ubuntu2
libapt-pkg4.12:amd64 1.0.1ubuntu2
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install ubuntu-minimal
[...]
0 upgraded, 61 newly installed, 0 to remove and 4 not upgraded.
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove ubuntu-minimal
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  ubuntu-minimal*
0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.
After this operation, 61.4 kB disk space will be freed.
Do you want to continue? [Y/n]

[Adam Conrad (adconrad)]

(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# dpkg -l apt apt-utils libapt-inst1.5 libapt-pkg4.12 | awk '/^.i/ {print $2 " " $3}'
apt 1.0.1ubuntu2.8
apt-utils 1.0.1ubuntu2.8
libapt-inst1.5:amd64 1.0.1ubuntu2.8
libapt-pkg4.12:amd64 1.0.1ubuntu2.8
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install ubuntu-minimal
[...]
0 upgraded, 61 newly installed, 0 to remove and 4 not upgraded.
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove ubuntu-minimal
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  cron* dmsetup* eject* init-system-helpers* iputils-ping* libbsd0*
  libcap2-bin* libdevmapper1.02.1* libestr0* libgcrypt11* libgnutls-openssl27*
  libgnutls26* libgpg-error0* libp11-kit0* libpam-cap* libtasn1-6* logrotate*
  lsb-release* netcat-openbsd* ntpdate* rsyslog* sudo* ubuntu-minimal* ucf*
  ureadahead* vim-common* vim-tiny*
0 upgraded, 0 newly installed, 27 to remove and 4 not upgraded.
After this operation, 9575 kB disk space will be freed.
Do you want to continue? [Y/n]

[Adam Conrad (adconrad)]

(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# dpkg -l apt apt-utils libapt-inst1.5 libapt-pkg4.12 | awk '/^.i/ {print $2 " " $3}'
apt 1.0.1ubuntu2.9
apt-utils 1.0.1ubuntu2.9
libapt-inst1.5:amd64 1.0.1ubuntu2.9
libapt-pkg4.12:amd64 1.0.1ubuntu2.9
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get install ubuntu-minimal
[...]
0 upgraded, 61 newly installed, 0 to remove and 0 not upgraded.
[...]
(trusty-amd64)root@nosferatu:/home/adconrad/apt-test# apt-get --purge autoremove ubuntu-minimal
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  ubuntu-minimal*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 61.4 kB disk space will be freed.
Do you want to continue? [Y/n]

[Adam Conrad (adconrad)]

Erm, and noticed my testing for bug #1429041 removed the wrong package on the "-notbroken" pass. Redid it on all three versions with removing meta instead of kernel, and it did correctly remove both packages, as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.1ubuntu2.9

---------------
apt (1.0.1ubuntu2.9) trusty; urgency=medium

  * Fix regression in the Never-MarkAuto-Sections feature caused by the
    previous auto-removal fix, with inspiration drawn from the patches
    and conversation from http://bugs.debian.org/793360 (LP: #1479207)

 -- Adam Conrad <email address hidden> Wed, 29 Jul 2015 21:23:49 -0600

[Chris J Arges (arges)]

Hello Adam, or anyone else affected,

Accepted apt into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.9.7ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Adam, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.9.10ubuntu6

---------------
apt (1.0.9.10ubuntu6) wily; urgency=medium

  * Fix regression in the Never-MarkAuto-Sections feature caused by the
    previous auto-removal fix, with inspiration drawn from the patches
    and conversation from http://bugs.debian.org/793360 (LP: #1479207)

 -- Adam Conrad <email address hidden> Sat, 01 Aug 2015 16:14:05 -0600

[Adam Conrad (adconrad)]

Verified on both precise and vivid.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.8.16~exp12ubuntu10.26

---------------
apt (0.8.16~exp12ubuntu10.26) precise; urgency=medium

  * Fix regression in the Never-MarkAuto-Sections feature caused by the
    previous auto-removal fix, with inspiration drawn from the patches
    and conversation from http://bugs.debian.org/793360 (LP: #1479207)

 -- Adam Conrad <email address hidden> Sat, 01 Aug 2015 16:18:59 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.9.7ubuntu4.2

---------------
apt (1.0.9.7ubuntu4.2) vivid; urgency=medium

  * Fix regression in the Never-MarkAuto-Sections feature caused by the
    previous auto-removal fix, with inspiration drawn from the patches
    and conversation from http://bugs.debian.org/793360 (LP: #1479207)

 -- Adam Conrad <email address hidden> Sat, 01 Aug 2015 16:14:56 -0600