Never-MarkAuto-Sections:: oldlibs gives wrong behavior

[Steve Langasek]

The /etc/apt/apt.conf.d/01autoremove file includes the following config:

  Never-MarkAuto-Sections
  {
[...]
        "oldlibs";
        "restricted/oldlibs";
        "universe/oldlibs";
        "multiverse/oldlibs";
  };

This was added back in 2010, with the following rationale:

message:
  add "oldlibs" to the APT::Never-MarkAuto-Sections as its used
  for transitional packages

This replaced a previous provisional 'transitional' section.

I don't understand exactly why this was done, but it seems very incorrect to me. The effect of Never-MarkAuto-Sections is to mark the *dependencies* of a package in this section as manually installed. This means that if you're using the historically accurate meaning of the 'oldlibs' section, any library dependencies of an old library will be marked manually installed, and thus will never be autoremoved when the old library will no longer be needed. E.g.:

$ apt-cache show libblas3gf
Package: libblas3gf
Priority: optional
Section: libs
Installed-Size: 40
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian Science Team <email address hidden>
Architecture: all
Source: blas
Version: 1.2.20110419-5
Depends: libblas3
Filename: pool/main/b/blas/libblas3gf_1.2.20110419-5_all.deb
Size: 2920
MD5sum: fb2afb44fdbdaf81d4adac5a509aac68
SHA1: be8056468c7b9668d2f22913853b8babc82af527
SHA256: 067cf1cdbb79372dfb838d534558cb6af29efc9d5d9765b639f9f39026dd2c42
Description-en: Transitional package for libblas
 Several minor changes to the C interface have been incorporated.
 One can maintain both versions on a system simultaneously to aid
 in the transition.
Homepage: http://www.netlib.org/blas/
Description-md5: 8bf7be122ddac6284a83fe69649495b3
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 18m
Task: ubuntu-usb, kubuntu-full, kubuntu-active-full, edubuntu-desktop-kde, edubuntu-desktop-gnome, edubuntu-usb, ubuntustudio-video, ubuntustudio-publishing, ubuntustudio-photography, ubuntustudio-graphics
$ apt-mark showmanual | grep libblas3
libblas3
libblas3gf
$

I think the *intent* was to ensure that when a package becomes a transitional package, it can then be removed without causing the real package that is the target of the transition to also be removed. But I don't think this is a correct interpretation of the 'oldlibs' section; packages in oldlibs are generally not leaf packages, and their dependencies are generally real dependencies rather than dependencies for purpose of a transition. Can we revisit this use of oldlibs?

FWIW, we seem to be doing a poor job in general of getting packages correctly marked for autoremoval. On my desktop system:

$ for pkg in $(apt-mark showmanual) ; do grep-status -FPackage -X $pkg -a -FSection -X libs -sPackage; done | uniq | wc -l
889
$

I don't think those are all due to this particular bug, but I'm pretty sure they're almost all wrong. :/

[Daniel Hartwig (wigs)]

On 6 November 2012 08:31, Steve Langasek <email address hidden> wrote:
> FWIW, we seem to be doing a poor job in general of getting packages
> correctly marked for autoremoval. On my desktop system:
>
> $ for pkg in $(apt-mark showmanual) ; do grep-status -FPackage -X $pkg -a -FSection -X libs -sPackage; done | uniq | wc -l
> 889
> $
>
> I don't think those are all due to this particular bug, but I'm pretty
> sure they're almost all wrong. :/

Potential sources of this:
- upgrading packages with some combinations of aptitude
  (< 0.6.4-1) and apt (≥ 0.8.11.2); [1]
- some other issues in current versions of aptitude (problem resolver;
  cancelling removal of packages);
- upgrading packages with python-aptdaemon (e.g. via
  software-center). [2]

These are just the cases that I am aware of. I have investigated some
others (apt, python-apt) which seem to be ok, though python-apt
conflates “FromUser=true” (pkgDepCache::MarkInstall et al.) with “mark
this package manually installed” in mark_upgrade, which is not
strictly true.

If the system is a relatively fresh install, then it is easier to
determine whether the issue is still present.

[1] http://bugs.debian.org/432017#152
[2] http://bugs.debian.org/685044

[Steve Langasek (vorlon)]

On Tue, Nov 06, 2012 at 02:03:50AM -0000, Daniel Hartwig wrote:
> On 6 November 2012 08:31, Steve Langasek <email address hidden> wrote:
> > FWIW, we seem to be doing a poor job in general of getting packages
> > correctly marked for autoremoval. On my desktop system:

> > $ for pkg in $(apt-mark showmanual) ; do grep-status -FPackage -X $pkg -a -FSection -X libs -sPackage; done | uniq | wc -l
> > 889
> > $

> > I don't think those are all due to this particular bug, but I'm pretty
> > sure they're almost all wrong. :/

> Potential sources of this:
> - upgrading packages with some combinations of aptitude
> (< 0.6.4-1) and apt (≥ 0.8.11.2); [1]
> - some other issues in current versions of aptitude (problem resolver;
> cancelling removal of packages);

Have never used aptitude on this system.

> - upgrading packages with python-aptdaemon (e.g. via
> software-center). [2]

I use update-manager for my upgrades, not software-center. Does the same
issue apply?

I think a lot of my issue probably comes to the way I reinstalled my system
to a new hard drive, using dpkg --get-selections rather than apt-clone. But
that doesn't explain it for any libraries that were installed after my
machine's brain-swap.

[Daniel Hartwig (wigs)]

On 7 November 2012 06:25, Steve Langasek <email address hidden> wrote:
>> - upgrading packages with python-aptdaemon (e.g. via
>> software-center). [2]
>
> I use update-manager for my upgrades, not software-center. Does the same
> issue apply?

Looking quickly at UpdateManager/backend/InstallBackendAptdaemon.py,
it's commit method does appear to request upgrades from aptdaemon (via
pkg_upgrades argument), which is where the problem is. Without
looking at it further, I'd say that the issue does apply.

The patch in the linked bug report shows clearly that the code is
wrong, although actually exposing the issue in tests took some time
(may be due to the workarounds in pkgDepCache::MarkInstall to "fix"
aptitude).

>
> I think a lot of my issue probably comes to the way I reinstalled my system
> to a new hard drive, using dpkg --get-selections rather than apt-clone.

There is that also.

[Daniel Hartwig (wigs)]

> I use update-manager for my upgrades, not software-center. Does the same
> issue apply?

The synaptic backend also causes trouble, since it uses a
--set-selections interface that, like dpkg, does not support keeping
packages marked auto-installed. See bug #300718 (update manager) and
bug #1075841 (synaptic).

Quickest solution for update-manager may be to apply the patch for
python-aptdaemon and drop the synaptic backend.

[Brian Murray (brian-murray)]

I've a system I've been upgrading since Feisty (or maybe Edgy), no reinstalling to a new hard drive silliness, and the output of Steve's command was: 1117.

I also use update-manager and not aptitude.

[Daniel Hartwig (wigs)]

On 14 November 2012 04:30, Brian Murray <email address hidden> wrote:
> I've a system I've been upgrading since Feisty (or maybe Edgy), no
> reinstalling to a new hard drive silliness, and the output of Steve's
> command was: 1117.
>
> I also use update-manager and not aptitude.

I have refiled the aptdaemon bug in launchpad, with a test case using
update-manager, since Debian doesn't particularly use the program.
Perhaps one of your Ubuntu guys could confirm the issue.

https://bugs.launchpad.net/bugs/1078544

[Daniel Hartwig (wigs)]

Agreeing with Steve, this is an obviously incorrect application of Never-MarkAuto-Sections.

[Julian Andres Klode (juliank)]

Marking this as released, as the behavior changed a while ago to "move" the autobit state of oldlibs on removal, rather than just marking the dependencies as manual; option APT::Move-Autobit-Sections in 01autoremove.