python 3.x compatibility issue

Bug #1184102 reported by Zoltán Halassy on 2013-05-25
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
pypolicyd-spf
Medium
Scott Kitterman

Bug Description

Have a 64bit Gentoo Hardened box, python 3.2 installed on it. I tried to get rid of python 2.7 but I just can't seem to make it work.

Used versions (system is pure 64bit):

Python 3.2.3
pypolicyd-spf 1.1.2
pyspf CVS 1.108.2.85 ( http://pymilter.cvs.sourceforge.net/viewvc/pymilter/pyspf/spf.py?revision=1.108.2.85&content-type=text%2Fplain )
pydns 3.0.1

I would like to know which pyspf version used for pypolicd-spf 1.1.2 testing on python 3.2, as the latest pyspf stable 2.0.7 is not python3 compatible (that's why i'm using a CVS version).

I think some of these bugs come from regression, but it would be cool, if we could nail them for python3. I'm happy to help, but I'm not a pyhton programmer, I don't have much experience.

The bugs I encountered are these:

1)

May 21 05:03:09 devon policyd-spf-python3.2[9238]: Traceback (most recent call last):
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/DNS/Base.py", line 181, in getSource
May 21 05:03:09 devon policyd-spf-python3.2[9238]: self.s.bind(('', source_port))
May 21 05:03:09 devon policyd-spf-python3.2[9238]: socket.error: [Errno 98] Address already in use
May 21 05:03:09 devon policyd-spf-python3.2[9238]:
May 21 05:03:09 devon policyd-spf-python3.2[9238]: During handling of the above exception, another exception occurred:
May 21 05:03:09 devon policyd-spf-python3.2[9238]: Traceback (most recent call last):
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/bin/policyd-spf-python3.2", line 684, in <module>
May 21 05:03:09 devon policyd-spf-python3.2[9238]: instance_dict, configData, peruser)
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/bin/policyd-spf-python3.2", line 530, in _spfcheck
May 21 05:03:09 devon policyd-spf-python3.2[9238]: res = spf.check2(ip, sender, helo)
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/spf.py", line 310, in check2
May 21 05:03:09 devon policyd-spf-python3.2[9238]: receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/spf.py", line 538, in check
May 21 05:03:09 devon policyd-spf-python3.2[9238]: spf = self.dns_spf(self.d)
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/spf.py", line 1094, in dns_spf
May 21 05:03:09 devon policyd-spf-python3.2[9238]: a = [t for t in self.dns_txt(domain) if RE_SPF.match(t)]
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/spf.py", line 1131, in dns_txt
May 21 05:03:09 devon policyd-spf-python3.2[9238]: for a in self.dns(domainname, 'TXT')]
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/spf.py", line 1255, in dns
May 21 05:03:09 devon policyd-spf-python3.2[9238]: for k, v in DNSLookup(name, qtype, self.strict, timeout):
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/spf.py", line 140, in DNSLookup
May 21 05:03:09 devon policyd-spf-python3.2[9238]: resp = req.req()
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/DNS/Base.py", line 227, in req
May 21 05:03:09 devon policyd-spf-python3.2[9238]: self.sendUDPRequest(server)
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/DNS/Base.py", line 253, in sendUDPRequest
May 21 05:03:09 devon policyd-spf-python3.2[9238]: self.conn()
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/DNS/Base.py", line 188, in conn
May 21 05:03:09 devon policyd-spf-python3.2[9238]: self.getSource()
May 21 05:03:09 devon policyd-spf-python3.2[9238]: File "/usr/lib64/python3.2/site-packages/DNS/Base.py", line 185, in getSource
May 21 05:03:09 devon policyd-spf-python3.2[9238]: if msg[0] != 98: raise
May 21 05:03:09 devon policyd-spf-python3.2[9238]: TypeError: 'error' object does not support indexing

2)

May 25 11:08:38 devon policyd-spf-python3.2[21264]: Traceback (most recent call last):
May 25 11:08:38 devon policyd-spf-python3.2[21264]: File "/usr/bin/policyd-spf-python3.2", line 684, in <module>
May 25 11:08:38 devon policyd-spf-python3.2[21264]: instance_dict, configData, peruser)
May 25 11:08:38 devon policyd-spf-python3.2[21264]: File "/usr/bin/policyd-spf-python3.2", line 420, in _spfcheck
May 25 11:08:38 devon policyd-spf-python3.2[21264]: foo = string.split(sender, '@', 1)
May 25 11:08:38 devon policyd-spf-python3.2[21264]: AttributeError: 'module' object has no attribute 'split'

Related branches

Zoltán Halassy (cf0hay) on 2013-05-25
description: updated

It's 2.0.7 with some patches (roughly equivalent to what's in CVS). Perhaps more importantly it's py3dns 3.0.2 and it did have an incompatible change from 3.0.1.

There is a known issue (as of yesterday) with IPv6 processing in pyspf on python 3 that I'm still trying to figure out.

I have figured out the issue I mentioned in the last comment.

If you look at http://anonscm.debian.org/viewvc/python-modules/packages/pyspf/tags/2.0.7-4/debian/patches/ you can see the two patches I've applied to pyspf for Debian/Ubuntu (including fixing the most recent issue).

I did go back and look and I'm pretty sure py3dns 3.0.1 won't work, you need 3.0.2 because of some incompatibilities between the two.

Scott Kitterman (kitterman) wrote :

pyspf 2.0.8 was just released that solves the python3.2 problems. It did, however, drop a function I was using in this package, so I need to release an update, which I will do shortly. py3dns 3.0.3, pyspf 2.0.8, and pypolicyd-spf 1.2 (not yet released) will solve this without any patching needed.

Scott Kitterman (kitterman) wrote :

1.2 is released. In combination with pyspf 2.0.8 it should work fine now.

Changed in pypolicyd-spf:
status: New → Fix Released
Download full text (4.5 KiB)

I'm still seeing the first of these errors after adding official python-3.x support to policyd-spf in Gentoo. We've got,

  * policyd-spf-1.2
  * pyspf-2.0.9
  * pydns-3.0.2
  * python-3.3.x or python-3.4.x

The machine on which I reproduced the error has python-3.3 built with python ipv6 support, but ipv6 disabled in the kernel. Below are the logs for one of the failed transactions.

May 10 00:19:10 mx1 postfix/smtpd[16219]: connect from smtprelay130c.careerbuilder.com[208.82.4.175]
May 10 00:19:10 mx1 postfix/smtpd[16219]: discarding EHLO keywords: DSN
May 10 00:19:10 mx1 policyd-spf[1301]: Starting
May 10 00:19:10 mx1 policyd-spf[1301]: Config: {'Mail_From_reject': 'Fail', 'Header_Type': 'SPF', 'debugLevel': 3, 'Mail_From_pass_restriction': 'spf_pass_from', 'defaultSeedOnly': 1, 'PermError_reject': 'False', 'skip_addresses': '127.0.0.0/8,::ffff:127.0.0.0/104,::1', 'Reject_Not_Pass_Domains': 'aexp.com,aim.com,aol.com,apple.com,dhl.com,gmail.com,googlemail.com,hallmark.com,hotmail.com,linkedin.com,live.com,msn.com,paypal.com,e.paypal.com,verizon.net', 'HELO_reject': 'Fail', 'TempError_Defer': 'False', 'HELO_pass_restriction': 'spf_pass_helo'}
May 10 00:19:10 mx1 policyd-spf[1301]: spfcheck: pyspf result: "['None', '', 'helo']"
May 10 00:19:10 mx1 policyd-spf[1301]: None; identity=helo; client-ip=208.82.4.175; helo=smtprelay130c.careerbuilder.com; <email address hidden>; <email address hidden>
May 10 00:19:10 mx1 policyd-spf[1301]: Traceback (most recent call last):
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib64/python3.3/site-packages/DNS/Base.py", line 192, in getSource
May 10 00:19:10 mx1 policyd-spf[1301]: self.s.bind(('', source_port))
May 10 00:19:10 mx1 policyd-spf[1301]: OSError: [Errno 98] Address already in use
May 10 00:19:10 mx1 policyd-spf[1301]:
May 10 00:19:10 mx1 policyd-spf[1301]: During handling of the above exception, another exception occurred:
May 10 00:19:10 mx1 policyd-spf[1301]: Traceback (most recent call last):
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib/python-exec/python3.3/policyd-spf", line 662, in <module>
May 10 00:19:10 mx1 policyd-spf[1301]: instance_dict, configData, peruser)
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib/python-exec/python3.3/policyd-spf", line 508, in _spfcheck
May 10 00:19:10 mx1 policyd-spf[1301]: res = spf.check2(ip, sender, helo)
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib64/python3.3/site-packages/spf.py", line 310, in check2
May 10 00:19:10 mx1 policyd-spf[1301]: receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib64/python3.3/site-packages/spf.py", line 555, in check
May 10 00:19:10 mx1 policyd-spf[1301]: spf = self.dns_spf(self.d)
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib64/python3.3/site-packages/spf.py", line 1120, in dns_spf
May 10 00:19:10 mx1 policyd-spf[1301]: a = [t for t in self.dns_txt(domain) if RE_SPF.match(t)]
May 10 00:19:10 mx1 policyd-spf[1301]: File "/usr/lib64/python3.3/site-packages/spf.py", line 1169, in dns_txt
May 10 00:19:10 mx1 policyd-spf[1301]: dns_list =...

Read more...

Scott Kitterman (kitterman) wrote :

I have occasionally seen this error as well:

May 10 00:19:10 mx1 policyd-spf[1301]: self.s.bind(('', source_port))
May 10 00:19:10 mx1 policyd-spf[1301]: OSError: [Errno 98] Address already in use

This is a py3dns/pydns error, not an error in pypolicyd-spf. I'll have a look at fixing it there.

Additionally, I just release pypolicyd-1.3 yesterday and once you have that with pyspf 2.0.9 you have an SPF system that's fully updated for the new SPF RFC, RFC 7208.

Scott Kitterman (kitterman) wrote :

This is fixed in py3dns 3.1.0 (you have 3.0.2). Here's the commit that fixed it:

http://bazaar.launchpad.net/~py3dns-hackers/py3dns/trunk/revision/121

Great, thanks. It looks like we missed the py3dns move to Launchpad and fell a few versions behind as a result. I'll get that fixed and require a newer pydns when I bump pypolicyd-spf.

Zoltán Halassy (cf0hay) wrote :

I'm still getting these:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.3/policyd-spf", line 680, in <module>
    instance_dict, configData, peruser)
  File "/usr/lib/python-exec/python3.3/policyd-spf", line 407, in _spfcheck
    foo = string.split(sender, '@', 1)
AttributeError: 'module' object has no attribute 'split'

Installed versions:

python 3.3
pyspf 2.0.9
pydns 3.1.0
pypolicyd-spf 1.3.1

Changed in pypolicyd-spf:
status: Fix Released → New
Zoltán Halassy (cf0hay) wrote :

I think replacing line 407 from this:

foo = string.split(sender,'@',1)

to this:

foo = sender.split('@',1)

would fix the problem

Zoltán Halassy (cf0hay) on 2014-08-26
summary: - 1.1.2 is broken on python 3.2
+ python 3.x compatibility issue
Scott Kitterman (kitterman) wrote :

Thanks, that's quite correct and I've fixed it for the next release.

More interesting (for me anyway) is why my test suite didn't catch it. I've fixed that as well.

Changed in pypolicyd-spf:
importance: Undecided → Medium
status: New → Fix Committed
assignee: nobody → Scott Kitterman (kitterman)
Scott Kitterman (kitterman) wrote :
Changed in pypolicyd-spf:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers