update-apt-xapian-index crashed with UnicodeDecodeError in record(): 'utf-8' codec can't decode byte 0xe3 in position 349: invalid continuation byte

[Gérson Soares Martins]

Errors Bucket
-------------
https://errors.ubuntu.com/problem/046d41d2bc088194bfeb0d785005fddfe64e6e6d

Não sei se se trata de "bug".

ProblemType: CrashDistroRelease: Ubuntu 16.04
Package: apt-xapian-index 0.47ubuntu7
ProcVersionSignature: Ubuntu 4.4.0-2.16-generic 4.4.0
Uname: Linux 4.4.0-2-generic i686
ApportVersion: 2.19.4-0ubuntu2
Architecture: i386
CrashCounter: 1
Date: Wed Feb 10 00:54:40 2016
ExecutablePath: /usr/sbin/update-apt-xapian-index
InstallationDate: Installed on 2011-04-28 (1748 days ago)InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release i386 (20101007)
InterpreterPath: /usr/bin/python3.5
PackageArchitecture: all
ProcCmdline: /usr/bin/python3 /usr/sbin/update-apt-xapian-index --quiet
PythonArgs: ['/usr/sbin/update-apt-xapian-index', '--quiet']SourcePackage: apt-xapian-index
Title: update-apt-xapian-index crashed with UnicodeDecodeError in record(): 'utf-8' codec can't decode byte 0xe3 in position 349: invalid continuation byte
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

[Gérson Soares Martins (1gerson.8)]

Não tenho mais informações, que talvez pudessem ajudar na correção desse problema...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[QkiZ (qkiz)]

Ubuntu 16.10, problem still exists

[QkiZ (qkiz)]

Problem is related to package apt-xapian-index. When I downloaded same package but from Debian repo and install it, after clearing /var/cache/apt-xapian-index and /var/lib/apt/lists/ I started update-apt-xapian-index again and everything pass ok. It looks like this problem is solved in Debian Jessie.

[Ulli Horlacher (framstag)]

This bug is still there in ubuntu 16.04.1!

root@diaspora:~# lsb_release -d
Description: Ubuntu 16.04.1 LTS

root@diaspora:~# update-apt-xapian-index -vf
(...)
Rebuilding Xapian index... 0%Traceback (most recent call last):
  File "/usr/sbin/update-apt-xapian-index", line 111, in <module>
    indexer.rebuild(opts.pkgfile)
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 758, in rebuild
    self.buildIndex(dbdir, generator)
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 733, in buildIndex
    for doc in documents:
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 580, in gen_documents_apt
    yield self.get_document_from_apt(pkg)
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 543, in get_document_from_apt
    addon.obj.index(document, pkg)
  File "/usr/share/apt-xapian-index/plugins/descriptions.py", line 108, in index
    self.indexer.index_text_without_positions(version.raw_description)
  File "/usr/lib/python3/dist-packages/apt/package.py", line 499, in raw_description
    return self._records.long_desc
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 48: invalid continuation byte

I found a workaround:

root@diaspora:~# LC_ALL=en_US.utf8 update-apt-xapian-index -vf
(...)
Writing value information to /var/lib/apt-xapian-index/values.
Writing prefix information to /var/lib/apt-xapian-index/prefixes.
Writing documentation to /var/lib/apt-xapian-index/README.
root@diaspora:~#

==> no more UTF errors!

(I also have to modify /etc/cron.weekly/apt-xapian-index !)

update-apt-xapian-index cannot handle non-UFT locale! I have:

root@diaspora:~# locale
LANG=en_US.ISO-8859-15
LANGUAGE=en_US:en
LC_CTYPE="en_US.ISO-8859-15"
LC_NUMERIC="en_US.ISO-8859-15"
LC_TIME=en_DK.UTF-8
LC_COLLATE="en_US.ISO-8859-15"
LC_MONETARY="en_US.ISO-8859-15"
LC_MESSAGES="en_US.ISO-8859-15"
LC_PAPER="en_US.ISO-8859-15"
LC_NAME="en_US.ISO-8859-15"
LC_ADDRESS="en_US.ISO-8859-15"
LC_TELEPHONE="en_US.ISO-8859-15"
LC_MEASUREMENT="en_US.ISO-8859-15"
LC_IDENTIFICATION="en_US.ISO-8859-15"
LC_ALL=

Switching my system completly to en_US.utf8 is NOT an option for me, for
several reasons.

[QkiZ (qkiz)]

16.04.1? Now it's 16.04.5. use package from Debian Jessie.

[Silvio Moioli (moio)]

17.04 and the problem is still there, using the package from Jessie works around the problem.

[Fenyvesi Attila (fattila2)]

I still have this problem with Ubuntu 16.04.3 LTS.

[Fenyvesi Attila (fattila2)]

root@HPp:/home/ata# update-apt-xapian-index -vf
Reading plugin /usr/share/apt-xapian-index/plugins/aliases.py.
Reading plugin /usr/share/apt-xapian-index/plugins/app-install.py.
Reading plugin /usr/share/apt-xapian-index/plugins/apttags.py.
Reading plugin /usr/share/apt-xapian-index/plugins/cataloged_time.py.
Reading plugin /usr/share/apt-xapian-index/plugins/debtags.py.
Reading plugin /usr/share/apt-xapian-index/plugins/descriptions.py.
Reading plugin /usr/share/apt-xapian-index/plugins/display_name.py.
Reading plugin /usr/share/apt-xapian-index/plugins/origin.py.
Reading plugin /usr/share/apt-xapian-index/plugins/relations.py.
Reading plugin /usr/share/apt-xapian-index/plugins/sections.py.
Reading plugin /usr/share/apt-xapian-index/plugins/sizes.py.
Reading plugin /usr/share/apt-xapian-index/plugins/software_center.py.
Reading plugin /usr/share/apt-xapian-index/plugins/template.py.
Reading plugin /usr/share/apt-xapian-index/plugins/translated-desc.py.
Most recent dataset: Sat Dec 2 12:15:33 2017.
Most recent update for: Sun Nov 12 12:45:46 2017.
Aggregating value information.
Initializing plugins.
Reading .desktop files from /usr/share/app-install/desktop/: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_yuri-sucupira_thunar1.6.10-fix_ubuntu_dists_xReading en translations from /var/lib/apt/lists/ppa.launchpad.net_yuri-sucupira_thunar1.6.10-fix_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading hu translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_i18n_TranslatReading hu translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_i18n_Translation-hu: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_i18n_TranReading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_i18n_Translation-en: done.
Reading hu translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_universe_i18n_TranslatioReading hu translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_universe_i18n_Translation-hu: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_i18n_Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_i18n_TReading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_i18n_Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_i18nReading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/...

[Fenyvesi Attila (fattila2)]

The workaround of Ulli Horlacher (framstag) doesn't work for me.

root@HPp:/home/ata# locale
LANG=hu_HU.UTF-8
LANGUAGE=hu:en
LC_CTYPE="hu_HU.UTF-8"
LC_NUMERIC=hu_HU.UTF-8
LC_TIME=hu_HU.UTF-8
LC_COLLATE="hu_HU.UTF-8"
LC_MONETARY=hu_HU.UTF-8
LC_MESSAGES="hu_HU.UTF-8"
LC_PAPER=hu_HU.UTF-8
LC_NAME=hu_HU.UTF-8
LC_ADDRESS=hu_HU.UTF-8
LC_TELEPHONE=hu_HU.UTF-8
LC_MEASUREMENT=hu_HU.UTF-8
LC_IDENTIFICATION=hu_HU.UTF-8
LC_ALL=

root@HPp:/home/ata# locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
hu_HU.utf8
POSIX

root@HPp:/home/ata# LC_ALL=en_US.utf8 update-apt-xapian-index -vf
Reading plugin /usr/share/apt-xapian-index/plugins/aliases.py.
Reading plugin /usr/share/apt-xapian-index/plugins/app-install.py.
Reading plugin /usr/share/apt-xapian-index/plugins/apttags.py.
Reading plugin /usr/share/apt-xapian-index/plugins/cataloged_time.py.
Reading plugin /usr/share/apt-xapian-index/plugins/debtags.py.
Reading plugin /usr/share/apt-xapian-index/plugins/descriptions.py.
Reading plugin /usr/share/apt-xapian-index/plugins/display_name.py.
Reading plugin /usr/share/apt-xapian-index/plugins/origin.py.
Reading plugin /usr/share/apt-xapian-index/plugins/relations.py.
Reading plugin /usr/share/apt-xapian-index/plugins/sections.py.
Reading plugin /usr/share/apt-xapian-index/plugins/sizes.py.
Reading plugin /usr/share/apt-xapian-index/plugins/software_center.py.
Reading plugin /usr/share/apt-xapian-index/plugins/template.py.
Reading plugin /usr/share/apt-xapian-index/plugins/translated-desc.py.
Most recent dataset: Sat Dec 2 12:15:33 2017.
Most recent update for: Sun Nov 12 12:45:46 2017.
Aggregating value information.
Initializing plugins.
Reading .desktop files from /usr/share/app-install/desktop/: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_yuri-sucupira_thunar1.6.10-fix_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading hu translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_i18n_Translation-hu: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_i18n_Translation-en: done.
Reading hu translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_universe_i18n_Translation-hu: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/ppa.launchpad.net_jonathonf_gcc-7.1_ubuntu_dists_xenial_main_i18n_Translation-en: done.
Reading en translations from /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_...

[Fenyvesi Attila (fattila2)]

Every time I turn my machine on, I get 2 or 3 dialogue windows saying "System program problem detected."
This is a bit annoying. Can anyone help me resolve this issue?

[QkiZ (qkiz)]

Did you try to install a version from Debian?

[Fenyvesi Attila (fattila2)]

I haven't tried the Debian version yet. I don't know how to do that but I will try and search for a guide.

[Fenyvesi Attila (fattila2)]

I installed version 0.49 from Debian. I had to purge version 0.47 (from Ubuntu) first. That way I could update the index. So the following steps worked for me: purge the Ubuntu package, then install the Debian version.

The answer of knarf on ubuntuforums.org says:
"The xapian index is used for the search box in synaptic. If you can live without that functionality you can purge the apt-xapian-index package"
https://ubuntuforums.org/showthread.php?t=1617639&p=10159339#post10159339

I found that I don't need the quick search function of synaptic provided by the apt-xapian-index package as much as I need the 145 MB of disk space that the index uses, so I removed the package.
The removal of the package affected only the quick search box. The search button is still there in synaptic and that is enough for me.

Thank you for your advice, QkiZ!