Hplip does not work with Python3: Unicode error

[Johan Myréen]

The hp-setup program crashes with a UnicodeDecodeError when it tries to detect my Photosmart 2575 printer. I am running Arch Linux which has switched to using Python3 by default (/usr/bin/python is a symlink to python3).

Below is a part of a debugging session demonstrating the problem. The variable x contains (part of) the response from my printer to the multicast query sent on the network. The to_string_utf8 function is supposed to convert the contents of the byte array to UTF-8, but chokes on the 0xf0 in position 364, presumably because the bytes aren't all strictly ASCII codepoints.

Running the program using Python2 does not trigger this bug.

My printer is a HP Photosmart 2575, Python version 3.4.3, Python2 version 2.7.10, Hplip version hplip 3.15.6-1 (Arch Linux).

(Pdb) n
> /usr/share/hplip/base/slp.py(110)detectNetworkDevices()
-> struct.unpack("!BBHBBHHHHH", data[:16])
(Pdb) n
> /usr/share/hplip/base/slp.py(112)detectNetworkDevices()
-> x = struct.unpack("!%ds" % attr_length, data[16:])[0].strip()
(Pdb) n
> /usr/share/hplip/base/slp.py(115)detectNetworkDevices()
-> x= to_string_utf8(x)
(Pdb) p x
b'(x-hp-ver=01)(x-hp-mac=00187100690d)(x-hp-num_port=01)(x-hp-ip=192.168.000.102)(x-hp-hn=Smarty)(x-hp-p1=MFG:HP;MDL:Photosmart 2570 series;CMD:MLC,PCL,PML,DW-PCL,DESKJET,DYN;1284.4DL:4d,4e,1;CLS:PRINTER;DES:6543;SN:MY66U1131C04DY;S:038000C484001021002c1800028c2880025;J: ;Z:0102,0503dea9015549,0600;BT:000000000000,50686F746F736D6172742032353\xd1\xbdZT\xf0!\x80i \x82\xc8,\xff\xff\xfa@\xf0,\x85-)(x-hp-guid=00187100690d)'
(Pdb) n
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 364: invalid continuation byte
> /usr/share/hplip/base/slp.py(115)detectNetworkDevices()
-> x= to_string_utf8(x)
(Pdb)

[Gaurav Sood (gaurav-sood)]

Please apply this patch and see if it works
1. Take a back up of /usr/share/hplip/base/sixext.py
1. Download this patch in any folder and navigate to that folder
2. Apply this patch as a root user using the following command

    'patch /usr/share/hplip/base/sixext.py < sixext.patch'

[Johan Myréen (r-ubuntuone-v)]

Yes, now the decode functions ignore the errors and the setup succeeds.

Thank you.

[Georges Toth (georges-toth)]

The latest Debian Stretch version is affected by this bug, which is version 3.16.11+repack0-2.

I ended up patching the "/usr/share/hplip/base/sixext.py" file exactly as in the suggested patch, which solved the issue for us.

Is there any information on when this quite old fix will make it into the upstream package?

[kritika arora (kritika-arora1)]

Hi,

The required changes has been included in HPLIP source and will be made available in the next release.

Thanks.

[Georges Toth (georges-toth)]

Hi,

Thank you!

[Daniel Pielmeier (daniel-pielmeier)]

According to #4 this has been fixed, however the same issue came up in Gentoo recently. Applying the patch from #1 fixed the issue.

What could be the problem here. Was this fix reverted or changed in another way. Maybe there is a regression somewhere.

[Francois Jaouen (francois-jaouen)]

Tested version 3.18.4 on Arch, the problem is still there. The proposed patch has not been applied :

% hp-probe -bnet
HP Linux Imaging and Printing System (ver. 3.18.4)
(...)
\Traceback (most recent call last):
  File "/usr/bin/hp-probe", line 162, in <module>
    devices = device.probeDevices([b], timeout, ttl, filter_dict, search, method)
  File "/usr/share/hplip/base/device.py", line 487, in probeDevices
    detected_devices = slp.detectNetworkDevices(ttl, timeout)
  File "/usr/share/hplip/base/slp.py", line 115, in detectNetworkDevices
    x= to_string_utf8(x)
  File "/usr/share/hplip/base/sixext.py", line 113, in to_string_utf8
    return s.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 365-366: invalid continuation byte

The printer is an HP Photosmart C6480 All in One.

[Francois Jaouen (francois-jaouen)]

Look like a duplicate of https://bugs.launchpad.net/hplip/+bug/1473066

[francois.jaouen (francois.jaouen-redhat-bugs)]

Description of problem:
Try to detect my network printer HP Photosmart C8260 using the command :
 % hp-probe -bnet -ldebug

Output of the command :

HP Linux Imaging and Printing System (ver. 3.18.4)
Printer Discovery Utility ver. 4.1

Copyright (c) 2001-15 HP Development Company, LP
This software comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to distribute it
under certain conditions. See COPYING file for more details.

--------------------
| DEVICE DISCOVERY |
--------------------

Probing network for printers. Please wait, this will take approx. 10 seconds...

hp-probe[1859]: debug: {}
hp-probe[1859]: debug: 0000: 01 07 01 a7 00 00 65 6e 00 03 00 00 00 00 01 97 28 78 2d 68 70 2d 76 65 72 3d 30 31 29 28 78 2d ......en........(x-hp-ver=01)(x-
hp-probe[1859]: debug: 0020: 68 70 2d 6d 61 63 3d 30 30 31 65 30 62 39 61 35 63 63 63 29 28 78 2d 68 70 2d 6e 75 6d 5f 70 6f hp-mac=001e0b9a5ccc)(x-hp-num_po
hp-probe[1859]: debug: 0040: 72 74 3d 30 31 29 28 78 2d 68 70 2d 69 70 3d 31 39 32 2e 31 36 38 2e 32 34 36 2e 30 31 31 29 28 rt=01)(x-hp-ip=192.168.246.011)(
hp-probe[1859]: debug: 0060: 78 2d 68 70 2d 68 6e 3d 48 50 39 41 35 43 43 43 29 28 78 2d 68 70 2d 70 31 3d 4d 46 47 3a 48 50 x-hp-hn=HP9A5CCC)(x-hp-p1=MFG:HP
hp-probe[1859]: debug: 0080: 3b 4d 44 4c 3a 50 68 6f 74 6f 73 6d 61 72 74 20 43 36 32 30 30 20 73 65 72 69 65 73 3b 43 4d 44 ;MDL:Photosmart C6200 series;CMD
hp-probe[1859]: debug: 00a0: 3a 4d 4c 43 2c 50 43 4c 2c 50 4d 4c 2c 44 57 2d 50 43 4c 2c 44 45 53 4b 4a 45 54 2c 44 59 4e 3b :MLC,PCL,PML,DW-PCL,DESKJET,DYN;
hp-probe[1859]: debug: 00c0: 31 32 38 34 2e 34 44 4c 3a 34 64 2c 34 65 2c 31 3b 43 4c 53 3a 50 52 49 4e 54 45 52 3b 44 45 53 1284.4DL:4d,4e,1;CLS:PRINTER;DES
hp-probe[1859]: debug: 00e0: 3a 43 43 39 38 38 3b 53 4e 3a 4d 59 37 43 49 48 35 31 30 50 30 35 30 36 3b 53 3a 30 33 38 30 38 :CC988;SN:MY7CIH510P0506;S:03808
hp-probe[1859]: debug: 0100: 38 43 34 38 34 30 30 31 30 32 31 30 30 37 38 63 62 30 30 30 30 30 34 31 62 38 30 34 32 37 34 36 8C4840010210078cb0000041b8042746
hp-probe[1859]: debug: 0120: 62 38 30 34 34 38 34 37 62 38 30 34 34 65 34 34 62 38 30 34 36 34 34 38 62 38 30 30 35 35 34 35 b8044847b8044e44b8046448b8005545
hp-probe[1859]: debug: 0140: 62 38 30 34 36 34 3b 5a 3a 30 31 30 32 2c 30 35 30 30 30 30 30 39 30 31 36 61 38 35 30 31 33 32 b80464;Z:0102,05000009016a850132
hp-probe[1859]: debug: 0160: 34 35 30 31 33 32 34 35 30 31 33 32 34 35 30 31 33 32 34 31 30 31 33 32 34 35 21 73 54 f0 a0 29 45013245013245013241013245!sT..)
hp-probe[1859]: debug: 0180: 4d 7d 21 73 55 ac ff ff fa 40 a0 39 c7 3f 29 28 78 2d 68 70 2d 67 75 69 64 3d 30 30 31 65 30 62 M}!sU....@.9.?)(x-hp-guid=001e0b
hp-probe[1859]: debug: 01a0: 39 61 35 63 63 63 29 9a5ccc)
Traceback (most recent call last):
  File "/usr/bin/hp-probe", line 162, in <module>
    devices = device.probeDevices([b], timeout, ttl, filter_dict, search, method)
  File "/usr/share/hplip/base/device.py", line 487, in probeDevices
    detected_devices = slp.detectNetworkDevices(ttl, timeout)
  File "/usr/share/hplip/base/slp.py", line 115, in detectNe...

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429138
File: backtrace

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429139
File: cgroup

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429140
File: cpuinfo

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429141
File: environ

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429142
File: mountinfo

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429143
File: namespaces

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429144
File: open_fds

[francois.jaouen (francois.jaouen-redhat-bugs)]

Created attachment 1429145
File: hp-probe.log

[johnthacker (johnthacker-redhat-bugs)]

Created attachment 1528767
Patch provided in Launchpad bug

Patch for /usr/share/hplip/base/sixext.py provided in the Launchpad bug for the same issue.

[johnthacker (johnthacker-redhat-bugs)]

This is exposed by the switch to using Python3 by default. It's a known bug in HPlip that other distributions have seen when they switched to Python3 by default. It is still present in Fedora 29 and the new HPlip. There's a patch that makes things work (but is probably not the Right Way) by ignoring the UTF-8 conversion errors. Using the patch, everything works for me (and people on other distros.)