Muon rebuilds search index very often and inefficient

[Norbert]

Steps to reproduce:
1. Install Muon on 16.04 LTS and launch it
2. Click "Check for updates"
3. Wait for "Rebuilding Search Index" to finish
4. After some time open Muon again (without 'apt-get update', 'aptitude update').
5. Get 'Rebuilding search index' message with background process /usr/sbin/update-apt-xapian-index and high CPU load and high temperature as the result.

Expected results:
Muon runs update-apt-xapian-index only when it's really needed. Muon allows user to search packages instantly after launch.

Actual results:
Muon rebuilds search index very often and inefficient. Muon does not allow user to search packages instantly after launch.

Note:
This bug is very similar to the Synaptic one (see https://bugs.launchpad.net/bugs/1685376).
It is caused by problems in apt-xapian-index (mainly https://bugs.launchpad.net/bugs/1612948).

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: muon 4:5.6.0-0ubuntu1.16.04.1
ProcVersionSignature: Ubuntu 4.4.0-104.127-generic 4.4.98
Uname: Linux 4.4.0-104-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
CurrentDesktop: MATE
Date: Sun Jan 7 19:20:12 2018
InstallationDate: Installed on 2014-02-07 (1429 days ago)
InstallationMedia: Ubuntu 12.04.4 LTS "Precise Pangolin" - Release amd64 (20140204)
SourcePackage: muon
UpgradeStatus: Upgraded to xenial on 2017-04-15 (266 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Norbert (nrbrtx)]

Its unbelievable - I opened Muon, tried to make search, then it Rebuilt index, I can't make new search query.
I closed Muon, entered failed (second) query. It shown results. But Rebuilding Search Index again.

Synaptic is broken, Muon is buggy. Sometimes I do not want to use aptitude/apt/apt-cache and other TUI. GNOME/Ubuntu-software is slow.

Where should I write to help fixing this bug in Muon / apt-xapian-index?

[Norbert (nrbrtx)]

Dear maxy, could you please look to this bug?

[Norbert (nrbrtx)]

Steps to reproduce:
1. Install Muon on Ubuntu 16.04 LTS and launch it
2. Click "Check for updates"
3. Wait for "Rebuilding Search Index" to finish
4. After some time open Muon again (without 'apt-get update', 'aptitude update').
5. Get 'Rebuilding search index' message with background process /usr/sbin/update-apt-xapian-index and high CPU load and high temperature as the result.

Expected results:
Muon runs update-apt-xapian-index only when it's really needed. Muon allows user to search packages instantly after launch.

Actual results:
Muon rebuilds search index very often and inefficient. Muon does not allow user to search packages instantly after launch.

Note:
This bug is very similar to the Synaptic one (see https://bugs.launchpad.net/bugs/1685376).
It is caused by problems in apt-xapian-index (mainly https://bugs.launchpad.net/bugs/1612948).

Downstream bug-report:
https://bugs.launchpad.net/bugs/1741767

[Carlo Vanini (silhusk)]

Muon checks the modification time of `/var/lib/apt-xapian-index/update-timestamp`, `/var/lib/dpkg/status` and `/var/cache/apt/pkgcache.bin`. Only if the timestamp of the index is older than any of the other two, then the index is rebuilt.

Something is updating/touching files more often than expected. Maybe unattended upgrades?

Could you please run in a shell
sudo update-apt-xapian-index --force
just to see if the index is built correctly or it fails somewhere.

[Norbert (nrbrtx)]

Thank you for reply, Carlo!

I have already disabled systemd timers for APT and removed unnattended-upgrades package.

So before running Muon I have the following dates:

$ ls -lt /var/lib/apt-xapian-index/update-timestamp /var/lib/dpkg/status /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 4812978 фев 19 01:49 /var/lib/dpkg/status
-rw-r--r-- 1 root root 49338939 фев 18 17:21 /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 0 фев 17 22:07 /var/lib/apt-xapian-index/update-timestamp

Then I started Muon and waited 5 minutes:

$ ls -lt /var/lib/apt-xapian-index/update-timestamp /var/lib/dpkg/status /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 4812978 фев 19 01:49 /var/lib/dpkg/status
-rw-r--r-- 1 root root 0 фев 18 17:21 /var/lib/apt-xapian-index/update-timestamp
-rw-r--r-- 1 root root 49338939 фев 18 17:21 /var/cache/apt/pkgcache.bin

then closed Muon.

After this I have interesting process in `ps aux | grep xapian`:

 /usr/bin/python3 /usr/share/apt-xapian-index/update-apt-xapian-index-dbus

I opened Muon again, waited 5 minutes and tried to search for `scilab` - it does not return results, after closing Muon I have:

$ ls -lt /var/lib/apt-xapian-index/update-timestamp /var/lib/dpkg/status /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 4812978 фев 19 01:49 /var/lib/dpkg/status
-rw-r--r-- 1 root root 0 фев 18 17:21 /var/lib/apt-xapian-index/update-timestamp
-rw-r--r-- 1 root root 49338939 фев 18 17:21 /var/cache/apt/pkgcache.bin

I run `sudo update-apt-xapian-index --force`:

$ sudo update-apt-xapian-index --force

The index /var/lib/apt-xapian-index is up to date, but rebuilding anyway as requested.
Reading .desktop files from /usr/share/app-install/desktop/: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_yannubuntu_boot-repair_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_webupd8team_java_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_outwiker-team_ppa_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_jonathonf_firefox-esr_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_ubuntu-desktop_ubuntu-make_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updat...

[Carlo Vanini (silhusk)]

Git commit e4f2a1f79741c930802acab7150fa6b96240bed6 by Carlo Vanini.
Committed on 22/02/2018 at 09:27.
Pushed by vanini into branch 'master'.

Do not check status file for timestamp

Summary:
Update-apt-xapian-index is organised in plugins, each representing a
data source for the index. Each of them reports a timestamp of the
latest change, and the updated index takes the maximum of them as its
own timestamp.

We were checking the time of `Dir::State::status`
(`/var/lib/dpkg/status`), which is not used in any plugin. When it was
newer than all data sources, we would continuously see the index as out
of date.

Reviewers: sitter

Reviewed By: sitter

Differential Revision: https://phabricator.kde.org/D10715

M +1 -8 src/backend.cpp

https://commits.kde.org/libqapt/e4f2a1f79741c930802acab7150fa6b96240bed6

[Norbert (nrbrtx)]

Launched Muon from terminal

$ muon
Failed to establish shared memory mapping, will fallback to private memory -- memory usage will increase
Invalid pixmap specified.
No frame loaded
No frame loaded
No frame loaded
No frame loaded
No frame loaded
No frame loaded
Couldn't find the releasechecker script ("/home/nikolay/.local/share", "/usr/share/mate", "/usr/local/share", "/usr/share")
No frame loaded
No frame loaded
No frame loaded
Search error "The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation"
Search error "The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation"
Search error "The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation"
Search error "The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation"

Then in GUI it rebuilds index.

[Norbert (nrbrtx)]

Also seen on 19.04 beta.

[S-zb (s-zb)]

I feel the need to resurrect this bug report since the problem seems to have resurfaced (actually quite a while ago).

SUMMARY
Muon rebuilds its search index almost every time it is opened, even if only minutes have passed since the last time it was in use. It also does so right after installing a package. This leaves the user unable to search for or install packages during the time the index is being updated (which for me can take up to 2 minutes at times).

STEPS TO REPRODUCE
1. Launch Muon.
2. Search Index is rebuilding as indicated in the lower right corner.
3. Searching for a package yields 0 results until the index has completely been updated.

Operating System: KDE neon 5.15
KDE Plasma Version: 5.15.4
KDE Frameworks Version: 5.57.0
Qt Version: 5.12.0
Kernel Version: 4.15.0-47-generic

[Xenoidaltu (xenoidaltu)]

I confirm it to be happening on Kubuntu 18.04 LTS
But Muon is flawless on KDE Debian Buster.

[Mitch Golden (mgolden)]

Are people still seeing this? I do not observe this behavior on Kubuntu 20.10 with muon 5.8.0. Could it have been a packaging issue just on Neon?

[Bug-janitor (bug-janitor)]

Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!

[S-zb (s-zb)]

Contrary to earlier experiences, I do not see this behaviour anymore with Muon 5.8.0 under KDE Neon 5.19.4 (based on Ubuntu 18.04) and 5.20.4 (based on Ubuntu 20.04).

[S-zb (s-zb)]

Apologies, I incorrectly set the status to 'reported' when in actuality the bug is most likely gone by now! Therefore setting the status to 'resolved, worksforme'.