iotop can raise UnicodeDecodeError because /proc/pid/status can have non-utf-8 data in Name

[Mingun]

Sometimes iotop fails with following error:

$ sudo iotop
Traceback (most recent call last):
  File "/usr/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 737, in main
    main_loop()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 727, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 620, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib/python3.8/curses/__init__.py", line 105, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 612, in run_iotop_window
    ui.run()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 188, in run
    self.refresh_display(iterations == 0, total, current,
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 476, in refresh_display
    lines = self.get_data()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 457, in get_data
    return list(map(format, processes))
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 432, in format
    cmdline = p.get_cmdline()
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 309, in get_cmdline
    proc_status = parse_proc_pid_status(self.pid)
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 211, in parse_proc_pid_status
    for line in open('/proc/%d/status' % pid):
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 20: invalid continuation byte

I found that /proc/pid/status can have an invalid UTF-8 symbol as part of Name, for example:

$ cat /proc/112413/status
Name: Инстанц�
Umask: 0002
State: S (sleeping)
Tgid: 112291
Ngid: 0
Pid: 112413
PPid: 112287
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 4096
Groups: 4 24 27 30 46 116 126 1000
NStgid: 112291
NSpid: 112413
NSpgid: 112287
NSsid: 15882
VmPeak: 11092600 kB
VmSize: 11092564 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 5985748 kB
VmRSS: 5971968 kB
RssAnon: 5963752 kB
RssFile: 8212 kB
RssShmem: 4 kB
VmData: 6228788 kB
VmStk: 136 kB
VmExe: 8 kB
VmLib: 23640 kB
VmPTE: 12460 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
Threads: 83
SigQ: 0/126975
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000004
SigIgn: 0000000000001000
SigCgt: 2000000181004ccf
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 571776
nonvoluntary_ctxt_switches: 10927

The attached patch fixes the problem.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: iotop 0.6-24-g733f3f8-1 [modified: usr/lib/python3/dist-packages/iotop/data.py]
ProcVersionSignature: Ubuntu 5.4.0-75.84-generic 5.4.119
Uname: Linux 5.4.0-75-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.18
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Fri Jun 18 15:19:11 2021
Dependencies:

InstallationDate: Installed on 2020-11-24 (205 days ago)
InstallationMedia: Ubuntu 18.04.3 LTS "Bionic Beaver" - Release amd64 (20190805)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=ru_RU.UTF-8
 SHELL=/bin/bash
SourcePackage: iotop
UpgradeStatus: Upgraded to focal on 2020-11-25 (205 days ago)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "fix-iotop-encoding-errors.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Sergio Durigan Junior (sergiodj)]

Thank you for your bug report.

I can confirm this bug. Here are the steps to reproduce it:

# lxc launch images:ubuntu/focal iotop-bug1932523
# lxc shell iotop-bug1932523
$ apt update && apt install -y iotop gcc
$ echo -e '#include <unistd.h>\nint main(){sleep(1000);return 0;}' | gcc -x c -o 'Инстанц�' -
$ ./Инстанц�
## <Type Control-Z here to put the program in the backgroun>
$ iotop
Traceback (most recent call last):
  File "/usr/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 737, in main
    main_loop()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 727, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 620, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib/python3.8/curses/__init__.py", line 105, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 612, in run_iotop_window
    ui.run()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 188, in run
    self.refresh_display(iterations == 0, total, current,
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 476, in refresh_display
    lines = self.get_data()
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 457, in get_data
    return list(map(format, processes))
  File "/usr/lib/python3/dist-packages/iotop/ui.py", line 432, in format
    cmdline = p.get_cmdline()
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 308, in get_cmdline
    proc_status = parse_proc_pid_status(self.pid)
  File "/usr/lib/python3/dist-packages/iotop/data.py", line 210, in parse_proc_pid_status
    for line in open('/proc/%d/status' % pid):
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 20: invalid continuation byte

I was able to verify that this bug still happens on impish, using a recent version of iotop (0.6-24-g733f3f8-1.1).

@Mingun, I think the best course of action for this bug is to contact upstream and suggest the patch to them first. Woud you be willing to do that, please?

Thank you.

[Paul Wise (Debian) (pabs)]

Hi folks,

Mingun: thanks for the patch, I would like to commit it upstream, but I
need to know what author name/email I should use for the commit and
what commit message to use.

If you know how to use git and already have a local commit, please use
the `git format-patch @{u}` command get a patch file with that info.

Once the commit author/message are available, I will commit the change
upstream and later I will update the Debian/Ubuntu package.

PS: until this gets fixed properly in Debian/Ubuntu, you might like to
use the iotop-c package from Debian/Ubuntu.

https://github.com/Tomas-M/iotop

--
bye,
pabs

https://bonedaddy.net/pabs3/

[Mingun (alexander-sergey)]

Hi Paul,

You can write any commit message that you find appropriate, I'm not familiar with common practices in the project and probably my English not so good. You also can made that commit without my attribution: this change is so simple that anyone can made it. I do not want to put progress of fixing bugs to the bureaucracy. But if you wish, you can use the following attributes:

name: Yanichkin Alexander
email: <email address hidden>

[Paul Wise (Debian) (pabs)]

Thanks, I have forward-ported the patch, added a commit message and pushed the patch upstream:

https://repo.or.cz/iotop.git/commit/443737ec620a699286b9b2e44dbcaac53f553812

At some point in the next few weeks I'll make a new release, upload it to Debian and get it synced to Ubuntu.

Since the patch is only a workaround, it would be good to have a proper fix; guess the encoding using chardet or similar and coerce the bytes into Unicode. Any help with that is welcome.

 status fixcommitted