policyd-spf dies with UnicodeDecodeError

[Michael Orlitzky]

This looks similar to bug 832480. In the logs, we have things like,

Jun 2 08:34:27 mx1 policyd-spf[22478]: Traceback (most recent call last):
Jun 2 08:34:27 mx1 policyd-spf[22478]: File "/usr/lib/python-exec/python3.3/policyd-spf", line 651, in <module>
Jun 2 08:34:27 mx1 policyd-spf[22478]: line = sys.stdin.readline()
Jun 2 08:34:27 mx1 policyd-spf[22478]: File "/usr/lib64/python3.3/encodings/ascii.py", line 26, in decode
Jun 2 08:34:27 mx1 policyd-spf[22478]: return codecs.ascii_decode(input, self.errors)[0]
Jun 2 08:34:27 mx1 policyd-spf[22478]: UnicodeDecodeError: 'ascii' codec can't decode byte 0x92 in position 241: ordinal not in range(128)
Jun 2 08:34:27 mx1 postfix/spawn[10886]: warning: command /usr/bin/policyd-spf exit status 1
Jun 2 08:34:27 mx1 postfix/smtpd[15342]: warning: premature end-of-input on private/policyd-spf while reading input attribute name
Jun 2 08:34:28 mx1 postfix/smtpd[15342]: warning: problem talking to server private/policyd-spf: Success
Jun 2 08:34:28 mx1 postfix/smtpd[15342]: NOQUEUE: reject: RCPT from 66-118-165-114.static.sagonet.net[66.118.165.114]: 451 4.3.5 Server configuration problem; from=<Rachel?<email address hidden>> to=<email address hidden> proto=ESMTP helo=<zka114.showedin.com>

All of these have been spam, so it's no great loss. I've tried increasing the DebugLevel to 5, but to no avail: nothing gets logged before the crash.

[Scott Kitterman (kitterman)]

I did investigate this issue. 832480 was about non-ascii characters in DNS records. This issue is about non-ascii in email addresses. The reason nothing gets logged is the crash is in the line that reads the data coming from postfix. The python readline function auto-detects the content type and it auto-detects ascii, which is wrong (thus the similar looking error).

It's extremely unlikely this will affect legitimate mail, but I believe it's possible. I'll get a fix out in the next day or two as it's taking some low level digging in the Python io module to try and sort out how to do this without the incorrect auto-detection.

Thanks for filing the bug.

[Scott Kitterman (kitterman)]

I believe I have this fixed. I have a few other issues to deal with as well as additional backward compatibility testing (only tested with python2.7 and 3.4 so far) before I can do a release, but if you want to try out the change, you can get the updated /usr/bin/policyd-spf at http://bazaar.launchpad.net/~kitterman/pypolicyd-spf/trunk/download/head:/42%40120a248f-6e16-0410-bfee-e5bfea1f629d:postfix-policyd-spf-python%252Ftrunk%252Fpolicyd-spf/policyd-spf

[Scott Kitterman (kitterman)]

Just released 1.3.1 to fix this problem.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pypolicyd-spf - 1.3.1-1

---------------
pypolicyd-spf (1.3.1-1) unstable; urgency=medium

  * Fix crashes caused by non-ascii characters in email addresses and lack
    of input validation (LP: #1325579)

 -- Scott Kitterman <email address hidden> Wed, 04 Jun 2014 02:24:50 -0400

[Michael Orlitzky (michael-orlitzky)]

Thanks! I ran it overnight, but none of Rachel's Health Tips have made it past postscreen yet. I'll keep an eye on it.

[Michael Orlitzky (michael-orlitzky)]

She's back, but at least I get a different error this time =)

Jun 16 12:17:57 mx1 policyd-spf[5112]: Traceback (most recent call last):
Jun 16 12:17:57 mx1 policyd-spf[5112]: File "/usr/lib/python-exec/python3.3/policyd-spf", line 654, in <module>
Jun 16 12:17:57 mx1 policyd-spf[5112]: line = lineraw.decode('UTF-8')
Jun 16 12:17:57 mx1 policyd-spf[5112]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 13: invalid start byte
Jun 16 12:17:57 mx1 postfix/spawn[5110]: warning: command /usr/bin/policyd-spf exit status 1
Jun 16 12:17:57 mx1 postfix/smtpd[5104]: warning: premature end-of-input on private/policyd-spf while reading input attribute name
Jun 16 12:17:58 mx1 postfix/smtpd[3202]: timeout after END-OF-MESSAGE from localhost[127.0.0.1]
Jun 16 12:17:58 mx1 postfix/smtpd[3202]: disconnect from localhost[127.0.0.1]
Jun 16 12:17:58 mx1 policyd-spf[7742]: Starting
Jun 16 12:17:58 mx1 policyd-spf[7742]: Traceback (most recent call last):
Jun 16 12:17:58 mx1 policyd-spf[7742]: File "/usr/lib/python-exec/python3.3/policyd-spf", line 654, in <module>
Jun 16 12:17:58 mx1 policyd-spf[7742]: line = lineraw.decode('UTF-8')
Jun 16 12:17:58 mx1 policyd-spf[7742]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 13: invalid start byte
Jun 16 12:17:58 mx1 postfix/spawn[27966]: warning: command /usr/bin/policyd-spf exit status 1
Jun 16 12:17:58 mx1 postfix/smtpd[5104]: warning: premature end-of-input on private/policyd-spf while reading input attribute name
Jun 16 12:17:58 mx1 postfix/smtpd[5104]: warning: problem talking to server private/policyd-spf: Success
Jun 16 12:17:58 mx1 postfix/smtpd[5104]: NOQUEUE: reject: RCPT from unknown[192.227.244.39]: 451 4.3.5 Server configuration problem; from=<Rachel?<email address hidden>> to=<email address hidden> proto=ESMTP helo=<popv39.thlaunch.com>