UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 24: invalid start byte

[nleo]

$ sudo aa-logprof
Reading log entries from /var/log/syslog.
Updating AppArmor profiles in /etc/apparmor.d.
Traceback (most recent call last):
  File "/usr/sbin/aa-logprof", line 52, in <module>
    apparmor.do_logprof_pass(logmark)
  File "/usr/lib/python3/dist-packages/apparmor/aa.py", line 2261, in do_logprof_pass
    handle_children('', '', root)
  File "/usr/lib/python3/dist-packages/apparmor/aa.py", line 1236, in handle_children
    sev_db.load_variables(profile)
  File "/usr/lib/python3/dist-packages/apparmor/severity.py", line 180, in load_variables
    for line in f_in:
  File "/usr/lib/python3.4/codecs.py", line 704, in __next__
    return next(self.reader)
  File "/usr/lib/python3.4/codecs.py", line 635, in __next__
    line = self.readline()

> dpkg -s apparmor
Package: apparmor
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 1181
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Version: 2.8.95~2430-0ubuntu5
Replaces: apparmor-parser, apparmor-utils (<< 2.8.0-0ubuntu28), libapache2-mod-apparmor (<< 2.5.1-0ubuntu3)
Depends: libc6 (>= 2.14), debconf (>= 0.5) | debconf-2.0, python3, libapparmor-perl, lsb-base, initramfs-tools, debconf
Suggests: apparmor-profiles, apparmor-docs, apparmor-utils
Breaks: apparmor-easyprof-ubuntu (<< 1.1.14), apparmor-utils (<< 2.8.0-0ubuntu28), libapache2-mod-apparmor (<< 2.5.1-0ubuntu3), libvirt-bin (<< 1.2.2-0ubuntu9~), lightdm (<< 1.9.14-0ubuntu2~), lxc (<< 1.0.2-0ubuntu2~)
Conffiles:

> cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=17
DISTRIB_CODENAME=qiana
DISTRIB_DESCRIPTION="Linux Mint 17 Qiana"

 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  File "/usr/lib/python3.4/codecs.py", line 548, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib/python3.4/codecs.py", line 494, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 24: invalid start byte

[Jamie Strandboge (jdstrand)]

What locale are you using? The version of the tools you are using also has a lot of bugs and the 2.9 release is much improved. Are you able to test this with a newer apparmor version? If not, knowing the locale will allow us to test. Thanks!

[Seth Arnold (seth-arnold)]

This is "new" with Python 3's switch from treating opened files as a binary stream to treating them as a formatted stream requiring codec support. Of course the logs can contain non-UTF8 characters, so treating them as UTF8 is wrong. I'm not sure what the fix will look like but I suspect it'll be convincing Python to just give us a stream of bytes as it used to.

[Ralf Spenneberg (ralq)]

This happens on Trusty Tahr 14.04 as well:

german installation

apparmor-utils 2.8.95~2430-0ubuntu5

[Christian Boltz (cboltz)]

Yes, charset problems are a real annoyance with python3 :-/

We chose to use errors='surrogateescape'.

Can you please test with AppArmor 2.9.0? It got lots of bugfixes, including a fix for this issue.

(Technically, this bug is fixed in upstream Apparmor, but an update for trusty is needed.)

[Ralf Spenneberg (ralq)]

It would be great to test 2.9.0 but unfortunately I am unable to install this version. What yacc is needed for libapparmor? None of the variants available on trusty tahr seems to work.

Using just the utils apparently does not work.

[Christian Boltz (cboltz)]

Sorry for the delay!

Replacing the utils and the python modules should be enough (most code is in the python modules, the aa-* tools are basically a wrapper around them ;-)

The easiest solution is probably to download the 2.9.x tarball (or do a bzr checkout of the 2.9 branch) and then
    cd $tarball/utils && python aa-logprof
which should use the modules from the tarball.

[Thomas E. Horner (thomas.horner)]

@Christian

root@www:/usr/local/src/apparmor-2.9.1/utils# python aa-logprof
Traceback (most recent call last):
  File "aa-logprof", line 18, in <module>
    import apparmor.aa as apparmor
  File "/usr/local/src/apparmor-2.9.1/utils/apparmor/aa.py", line 29, in <module>
    import apparmor.logparser
  File "/usr/local/src/apparmor-2.9.1/utils/apparmor/logparser.py", line 18, in <module>
    import LibAppArmor
ImportError: No module named LibAppArmor

[Thomas E. Horner (thomas.horner)]

above error is solved by
apt-get install python-libapparmor

[Thomas E. Horner (thomas.horner)]

to make things more convenient:
alias aa-logprof='pushd /usr/local/src/apparmor-2.9.1/utils >/dev/null; /usr/bin/python aa-logprof; popd >/dev/null'

[Christian Boltz (cboltz)]

I'm quite sure this bug is solved since quite a while.

Please reopen if you can reproduce it with AppArmor 2.9.1 or newer.