System cleaner removes explicitly installed third-party packages

[Lechuan]

Binary package hint: system-cleaner

Description: Ubuntu intrepid (development branch)
Release: 8.10

Package version: system-cleaner-1.10.3-0ubuntu1

Expected behavior: respect user decision and preserve explicitly installed third-party packages

What happened instead:

The way system-cleaner identify some packages as "cruft" is problematic and worrying. I got my Chinese input method removed by system-cleaner, and when I reinstalled the two packages using dpkg, they are listed by system-cleaner as "cruft" immediately.

The said packages are for SunPinyin, which are not in repo, and can be found here: http://blog.eshangrao.com/2008/08/25/555/.
I'm also posting the direct links here as most of you probably can't read Chinese:
http://blog.eshangrao.com/wp-content/uploads/file/scim-sunpinyin_1.0~hg20080607-2_i386.deb
and
http://blog.eshangrao.com/wp-content/uploads/file/sunpinyin-data-le_1.0~hg20080607-2_all.deb

I tested again with Opera (opera_9.60.2444.gcc4.qt3_i386.deb provided by Opera for Hardy), and immediately after installation it was listed as "cruft".

This is going a bit too far, and without warning, can cause quite a lot of agony on a normal user's part.

Propose fix: at least uncheck these explicitly installed third party packages by default.

SUMMARY FOR SRU: system-cleaner has no way of knowing whether a package is really obsolete or was installed via dpkg -i, or came from a repository no longer in sources.list. The workaround implemented is to add a dialog to have the user confirm they really want to remove packages.

PATCH: http://bazaar.launchpad.net/~systemcleaner-hackers/systemcleaner/intrepid-sru-1-proposal/changes (revision 111).

TEST CASE: Add package from third-party repository, remove repository from sources.list, then verify that system-cleaner-gtk offers to remove the package, and pops up the confirmation dialog.

[to be removed (liw)]

Unfortunately, there is currently no way for system-cleaner to know which packages are obsolete (because they are no longer available via apt-get), and which ones are third-party packages installed outside of apt. Thus the bug is valid, but we can't fix it for now.

I discussed this with Michael Vogt. We'll work on changing libapt after intrepid to make this distinction possible in the future.

[Lechuan (lechuan)]

Nice and fair. I propose though that system-cleaner come with some kind of warning so that users will know better than just click away and regret.

[Fernando Miguel (fernandomiguel)]

In my case it tried to remove webmin.

I know a lot of ppl dont like it, but it gets done I need.

[Chris Jones (cmsj)]

It's been discussed elsewhere that tracking packages deeply enough to differentiate between locally installed ones and crufty ones is quite tricky, and should be done later.

I spoke to Lars and suggested that in the short term (probably as a SRU after release) there be a whitelist of commonly installed third party packages. Lars suggested that there be a conf.d, which I think is a great idea. This will allow him to supply a list of commonly installed packages, but will also make it really easy for people to add their own, so you'll be able to stop the cleanup tool from removing your favourite third party software.

[Peter Jolly (peter-jollys)]

In the meantime, perhaps the message in the Cruft Remover window could be changed from "This application helps you to get rid of cruft" to something clearer along the lines of "This application helps you to identify and remove packages that might no longer be necessary. WARNING: it is not currently possible to tell the difference between obsolete Ubuntu packages and third-party packages that you installed yourself."

(The existing message is rather unhelpful in any case -- "cruft" is hacker jargon that will confuse many users, and should not be used without explanation.)

Even if it's too late to change this in all the translations, just doing this in English would be better than nothing.

[Bernhard (b.a.koenig)]

<translations>: To avoid the translation issue for the warning, you could put up a picture of Freddie Krueger and call it "crypt remover". :-)

But seriously, every user can go through the packages suggested by the remover and uncheck those that are not to be removed. Or maybe all packages could be unchecked by default?

[Daniel Queirolo (danf-1979)]

This software is not ready for production. You open the system-cleaner and you see a bunch of packages marked. No info is displayed, you don't know what the "marked" status mean, and if you decide to press the cleanup button, bye bye packages! no warning, no nada, cero.

As someone said on IRC:

<Daisuke_Ido> it's even worse for someone who comes over from windows and installs software from outside sources. they go to clean things up and all of their custom-installed software is gone.

I say that in present state, this software go against bug number 1. Please remove this from ubuntu-desktop asap.

[Gabriel Bauman (gabrielbauman)]

Besides all of the above, auto-removable packages don't turn up in this tool. Really, auto-removable packages should be all that's represented here until we can figure out how to avoid removing intentionally-installed software.

Idea 1:

Why not modify gdebi to record which out-of-repository packages the user installs? That way we could avoid removing those packages, since we know they were explicitly added by the user.

Idea 2:

Add a "Not Cruft" button to cruft-remover. This should write to the same 'whitelist' gdebi would contribute to in idea 1.

[MDylanBell (m-dylan-bell)]

I think it is a really bad idea to include this, in its current form, by default in Intrepid. It seems from the name like a typical system-cleaner app that one might find and use for windows, but it could actually harm someone's user experience by removing software they have installed.

I really, really don't like that this is included by default, since it really doesn't do anything useful yet and will confuse new users (or even experienced users who aren't paying attention).

[Tux (peter-hoogkamer)]

I agree. Just installed Intrepid Ibex RC on Vmware Workstation 5.5.0 and started system-cleaner right after boot. It shows a list of packages which really cannot be removed. If common users use this tool they will destroy their system :-( !!

[SK (stephantom)]

system-cleaner has been removed from the ubuntu-desktop dependencies.

[to be removed (liw)]

As Stephan wrote, system-cleaner (the package, the software having been renamed to Cruft Remover) has been removed from the default installation, for now. Much as it pains to admit my failure, I think this was the right decision, and I was one of the people suggesting this.

However, I'd still like to get Cruft Remover into a good state so that it can be installed by default later. After the intrepid release, I'll start work on that, and if you have any ideas how to improve that, please don't hesitate to add them to this bug, to file other bugs against the package, or to e-mail me directly (although I prefer public bug reports so that everyone can participate).

[Daniel Queirolo (danf-1979)]

Hi Lars, no failure, I read the wiki and the basic idea is ok, its just not the right time. Some ideas to improve the user experiencie:

1) Tooltip when hovering a package displays some information like the one line package description.
2) Make sure the user knows what the "checked" state means (that the package will be removed).
3) Ask a confirmation after the user press the clean button saying something like "10 packages are going to be removed, do you want to continue?", or something.
4) Someone said system-cleaner doesn't use --purge when removing packages, so the Cruft Remover leaves cruft behind.

[Bernhard (b.a.koenig)]

Yes, I have seen worse failures. :)
I would still like to see something like system-cleaner on my system. I guess many people just thought that it should be made clear to the users that this might damage their system. Hence give more warnings.

My question would be why this "system-cleaner/cruft remover" is launched independently of "apt-get autoremove" and "apt-get autoclean"? Why not incorporate everything into these two commands? And maybe have a GUI for them?

[Jags Desai (jagsdesai)]

Ubuntu Intrepid 8.10 (installed from i386 Desktop/Live CD) System Cleaner / Cruft Remover removes all the packages that were NOT installed by Synaptic:

1, OpenOffice.org 3.0
2, Solaris Nimbus theme
3, TrueCrypt

And not just "the boxes, that held the installed programs" but they were gone/removed completely from Main Menu, AWN & from Themes (Appearance Preferences) too.

I repeated installations/removals three times and all the time Cruft Remover removed all three installations completely. Cruft Remover screenshot attached.

I think the importance should be CRITICAL as in this related bug# 290024: https://bugs.launchpad.net/ubuntu/+source/system-cleaner/+bug/290024

[Dimitrios Symeonidis (azimout)]

I agree with more warnings.

I am wondering about the distinction between packages no longer in the repositories and packages installed directly (from the .deb):
can't we just search the repos of previous releases and see if the package has ever existed in them? If it has, then it's meant to be removed. Otherwise, it was installed by hand and it should probably stay (however, I would still put it in the list, with the tick turned off, so users see it and can make a choice)

[Jags Desai (jagsdesai)]

Warnings ??

Shouldn't System Cleaner / Cruft Remover be just "cleaning" unwanted stuff instead of removing/deleting packages/apps that are NOT unwanted. And it removed 'em even when they were in use... like Nimbus theme and OpenOffice Writer but still CR removed 'em.

[stlubuntu (jlrbennett)]

It amazes me the duplicative and competitive projects that occur in the FOSS community.
A prime example is the gnome nm applet and wicd. Another is ubuntu-eee and eeebuntu.
Again another is this Cruft remover and Ubucleaner. The author of Ubucleaner has already
successful created a script that performs this function correctly. A copy of this script
is attached. A web link is as follows:

http://www.opendesktop.org/content/show.php/Ubucleaner?content=71529

Since FOSS is free, distributable, and modifiable, the only reason for this repeated
disregard for others' work who have correctly solved a problem is pride.

Hope this helps. Please review the script. To me it is clear that this is what Cruft remover
intends to do.

[Gabriel Bauman (gabrielbauman)]

@stlubuntu:

Ubucleaner is a simplistic, non-user-friendly shell script that runs some simple apt commands and deletes trash folders with no user interaction. It runs three fixed aptitude commands and removes some folders - and that's it.

Cruft-cleaner is targeted to less technically-oriented users uncomfortable with the command line, has a user interface, and lets users make decisions on which packages are removed. It goes further than simply telling apt to clean up its cache and purge old configuration.

One is a five minute hack for power users, and the other is aiming to be a useful tool for everyone.

This comment area is for the discussion of a big in system-cleaner/cruft-remover. Please keep future discussion on-topic.

[stlubuntu (jlrbennett)]

Sorry, all I meant to suggest was that, in the FOSS spirit of standing on the shoulders of giants
or even little guys in this case, for a Cruft-cleaner that performs as the end users (everyone, not
just power users) would like, all that would need to happen is to code a gui for a utility or script based on
ubucleaner or at least incorporating the same functions. It just seemed to me that so much more could
be accomplished by building on existing code/script instead of starting from scratch.

That's all. Best Regards.

[Bernhard (b.a.koenig)]

<in the FOSS spirit of standing on the shoulders of giants or even little guys in this case> FOSS (vs Microsoft) spirit should be: giants are standing on our shoulders!

[Ævar Arnfjörð Bjarmason (avarab)]

I understand the inherent difficulty in determining if a package is a manually installed third-party package or obsolete as discussed earlier in this thread. One additional detection the tool might want to do in the future if and when it gets included in a release again is to check if the package is currently being used or vital to the system. The attached screenshots shows it suggesting that I remove my currently running kernel.

[to be removed (liw)]

I've made new packages available at

http://code.liw.fi/ubuntu/pool/main/s/system-cleaner/

They are supposed to improve the situation with regards to third party packages. Unfortunately, a proper fix isn't possible for intrepid, but we'll work on that for jaunty.

I would appreciate any feedback on whether the changes make the situation in intrepid sufficiently bearable. The primary fix is to add a new warning dialog after the user clicks on 'Cleanup', so that it is less easy to remove packages by mistake.

[Alan Pope 🍺🐧🐱 🦄 (popey)]

I have downloaded and installed the new packages from liw.fi and created a personal whitelist in /etc/cruft-remover.d/

alan@erasmus:/etc/cruft-remover.d$ cat personal.whitelist
deb:zattoo
deb:gitso
deb:virtualbox-2.0
deb:nautilus-dropbox

I can report that the system cleaner tool does indeed ignore those debs, and on this laptop I now get no packages listed for removal.

[Dean Loros (autocrosser)]

Thank you for the info--I have created the white-list & can report that Cruft-Remover now ignores my extra debs also...works very well.

deb:godesk
deb:lightscribe
deb:lightscribeapplications
deb:4l
deb:bluemarine

[Daniel Queirolo (danf-1979)]

Lars, when is this coming to jaunty?

[Åskar (olskar)]

The package is now default in Jaunty but this bug is not fixed. It still wants to remove third-party packages?

Perhaps a way would be to let manually installed packages automatically get into the whitelist but as of now, it should not be default IMO.

[Andrew Starr-Bochicchio (andrewsomething)]

I don't think that this can really be considered fixed until the feature is exposed to the GUI. A novice user who will possibly end up removing packages that they installed on purpose will not know to read the man page and create a "personal whitelist."

[James Hogarth (james-hogarth)]

The 'whitelist' is created when the packages are unticked.... although it is not clear that is the case.

Having computer-janitor alone wouldn't be a problem as far as I can see. For those that can make use of it's purpose but would rather not use synaptic can use the CLI for it.... Having computer-janitor-gtk in makes it much more likely that a manually installed package or more than one previous kernel would get removed.... But as per my comments on the forums I fail to see why this is a default package anyway and linked to ubuntu-desktop... It is power users only due to this issue.... power users are more likely to use synaptic anyway... power users can install from repos if needed (or OEMs etc)....

As per the notes above this was pulled from Intrepid as a default package being considered to risky due to the issue being discussed and that the issue would be worked on for jaunty.... The package was given a new name but the underlying issue still existed and being exposed to new users in a default menu option with a 'comfortable' GUI just compounds matters...

[to be removed (liw)]

The 'whitelist' is actually not created when the packages are unticked: however, Computer Janitor does remove the unticked state, so that a particular package only needs to be unticked once. This state is tracked in /var/lib/computer-janitor/state.dat.

The whitelist is files in /etc/computer-janitor.d, and those files need to be edited manually, for now.

Unfortunately, we didn't come up with a fix for apt/dpkg to fix this properly in jaunty. The basic problem is still that there is no reliable way to track what packages are really obsolete and what packages were not installed from repositories. Because there are a fair number of packages that get obsoleted from release to release, I am reluctant to disable this feature of Computer Janitor, despite all the false positives it generates.

That doesn't mean I'm ignoring the problem, just that it's not entirely easy for me to solve. If you think this part of CJ should be disabled for jaunty, could you start a discussion about it on the ubuntu-devel mailing list, to get more people pay attention this question?

[Bernhard (b.a.koenig)]

Btw, I currently have deluge installed from the official ppa archive but it's also suggested by cruft remover to be removed. This is something I don't understand. I think it should not suggest to remove packages just because I downloaded them from an "unofficial" archive.

[Bernhard (b.a.koenig)]

Aha, I just noticed that cruft remover suggests to remove a package if you downloaded it from an archive and then remove that later from your sources, eg:

1) add the deluge ppa to your sources
2) install deluge-torrent (then deluge-torrent is not considered cruft at that point)
3) now remove the deluge ppa from your sources, maybe because you don't want to check for new updates at this point because of a slow internet connection (then deluge-torrent is considered cruft)

I find this behavior unintuitive, even though I don't know what the "right" behavior would be. I guess that depends on how you implement this "whitelist" feature.

[to be removed (liw)]

Bernhard, yes, that's pretty much exactly what happens: if a package is no longer available in the package sources apt has, it's considered cruft. That's the best heuristic we have.

The whitelist is already implemented, and has been for a while. It is maintained manually by the user.

The state of each package is remembered, so that you only need to tell Computer janitor once that you don't want a package removed.

[Åskar (olskar)]

It might be the best heuristic we have at the moment but fact remains, it is not acceptable behavior. Recently a friend of mine removed a lot of manually installed packages thinking it was old settings for those programs or something. A lot of people will suffer from this.

Until this is fixed, I think this should be removed.

[phenest (steve-clark)]

This white list needs to be automated. That is to say, installing or removing 3rd party packages via Package Installer or dpkg should automatically update the white list. This would be in addition to unchecking/checking any packages in Computer Janitor.

[to be removed (liw)]

phenest, yes, the white list needs to be automated. This requires changes to dpkg, which did not happen during jaunty, unfortunately. Computer-janitor can't automate it, also unfortunately.

[phenest (steve-clark)]

I've always used Synaptic Package Manager for this purpose. One thing it does, is separate 'local or obsolete' from 'auto removable'. Can/does CJ do this?

[phenest (steve-clark)]

A further thought is the 'residual config' that Synaptic provides. This would be a good addition for CJ. Perhaps you should have some buttons to switch between the different types of cruft to remove.

[Jan Claeys (janc)]

I think the suggestion in comment #8 to change 'gdebi' so that it whitelists packages that are installed by it would help a lot?

[to be removed (liw)]

phenest, the "residual config" was planned for jaunty, but didn't make it, so it'll happen for karmic instead. It is a good idea, although not entirely risk-less.

CJ already separates between 'local or obsolete' and 'autoremovable'.

Jan, changing gdebi to whitelist packages would be a nice first step. It's not a complete solution, though, since not everyone uses gdebi. But it would be a step.

[NoOp (glgxg)]

And please modify to have all listed packaged unticked by default.

[alexey-t (alexey-t)]

Before the explicitly-installed package tracking is implemented, how about changing the wording in the GUI to make it more clear that packages will be *uninstalled*, not just package files removed? Current GUI reads (to me, at least) as if CJ will just remove .deb files that were used to install some software, and removal of the software itself comes as a surprise.

[Bernhard (b.a.koenig)]

Btw, I had another idea regarding the GUI, why not start the GUI without administrator mode (no password prompt) and then show a suggestions for removable packages. If the user really chooses to clean the suggested packages only then there is a warning and password prompt. This should give the user another idea that he is doing something to his system and it's another chance to stop the process.

[Carroarmato0 (carroarmato0)]

I'm am currently testing Karmic Koala Alpha 3, and this bug still suggests that the Flash Plugin, which was installed willingly from a third-party be removed. Now I know you might say that removing it might be a good idea in the long run ( ;) ), but actually trying to remove it causes some annoyances in the Package Management system. This being another bug from this one which I report separately.

But yeah, untill of Alpha 3 of Karmic, the Computer-janitor still makes some false assumptions of which package might be a good idea to remove.

[to be removed (liw)]

I've committed a change to make Computer Janitor not remove obsolete packages unless they are also automatically removable. This will make it miss a number of packages that probably should be cleaned up, but will more importantly reduce the number of false positives.

[Launchpad Janitor (janitor)]

This bug was fixed in the package computer-janitor - 1.13.2-0ubuntu1

---------------
computer-janitor (1.13.2-0ubuntu1) karmic; urgency=low

  * New upstream version. Closes bugs in Launchpad:
    - System cleaner removes explicitly installed third-party packages
      (LP: #285746)
    - I do not want to remove '/' (LP: #337241)
    - computer-janitor incomplete error description (LP: #365325)
    - computer-janitor commandline interface not pipe-friendly (LP: #375094)
    - Python 2.6 DeprecationWarning: Accessed deprecated property
      Package.candidateDownloadable (LP: #407127)
    - Should use GtkBuilder rather than libglade (LP: #403537)
    - Ambiguous wording in confirmation alert box (LP: #349336)
    - computer-janitor-gtk crashed with UnicodeDecodeError in
      get_description() (LP: #352461)
    - Can't deselect fixes with space in computer-janitor (LP: #355535)
    - relatime tweak will not be necessary in karmic (LP: #369151)
    - computer janitor needs filesize column (LP: #396522)
  * Packaging now uses dh from debhelper 7.

 -- Lars Wirzenius <email address hidden> Fri, 21 Aug 2009 15:48:12 +0300

[Bernhard (b.a.koenig)]

By the way, my Computer Janitor has just removed "software-store".

[Andrew Starr-Bochicchio (andrewsomething)]

On Mon, Sep 28, 2009 at 6:39 PM, Bernhard <email address hidden> wrote:
> By the way, my Computer Janitor has just removed "software-store".

That's correct. The package has been renamed software-center.
software-store is now a transitional dummy package that is safe to
remove.

[Steve Langasek (vorlon)]

On Mon, Sep 28, 2009 at 10:49:55PM -0000, Andrew Starr-Bochicchio wrote:
> That's correct. The package has been renamed software-center.
> software-store is now a transitional dummy package that is safe to
> remove.

No, it isn't; software-store has been removed altogether, there is no dummy
package. :)

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Bernhard (b.a.koenig)]

Maybe we should simply wait a little and "software-center" will probly be automatically installed in one of the future updates.

[Andrew Starr-Bochicchio (andrewsomething)]

On Mon, Sep 28, 2009 at 7:00 PM, Steve Langasek
<email address hidden> wrote:
> On Mon, Sep 28, 2009 at 10:49:55PM -0000, Andrew Starr-Bochicchio wrote:
>> That's correct. The package has been renamed software-center.
>> software-store is now a transitional dummy package that is safe to
>> remove.
>
> No, it isn't; software-store has been removed altogether, there is no dummy
> package. :)
>

Oh well, you know what they say about assuming.... =) I suppose as
software-store was never in a real release, there wasn't the need for
a transitional package.

Either way, it's not a bug with computer-janitor. software-store would
have been pulled in by its relationship to ubuntu-desktop. As it isn't
a dependency anymore, it's "apt-get autoremove"-able.