python-policyd-spf v1.3.2 + postfix v3.1.0, "Header_Type=SPF" works, but "Header_Type=AR" fails

[jasonsuser]

On a Postfix 3.1 svr, I've setup for use of python-policyd-spf 1.3.2

With

 Header_Type = SPF

in /etc/python-policyd-spf/policyd-spf.conf config, it all works exactly as you'd expect.

Checks are executed, PASS & FAIL work ok, & mail's tagged with the 'Received-SPF' header.

If I change only to the new style AR header (https://tools.ietf.org/html/rfc7001)

 /etc/python-policyd-spf/policyd-spf.conf

- Header_Type = SPF
+ Header_Type = AR
+ Authserv_Id = mail01.example.com

and reload postfix, the same test mail is now rejected

 Apr 3 11:22:58 mail01 postfix/postscreen[7388]: CONNECT from [209.85.192.195]:33458 to [192.0.2.23]:25
 Apr 3 11:22:58 mail01 postfix/dnsblog[7391]: addr 209.85.192.195 listed by domain list.dnswl.org as 127.0.5.1
 Apr 3 11:22:58 mail01 postfix/postscreen[7388]: PASS OLD [209.85.192.195]:33458
 Apr 3 11:22:58 mail01 postfix/psint/smtpd[7394]: connect from mail-pf0-f195.google.com[209.85.192.195]
 Apr 3 11:22:59 mail01 postfix/spawn[7398]: warning: command /usr/bin/python exit status 1
 Apr 3 11:22:59 mail01 postfix/psint/smtpd[7394]: warning: premature end-of-input on private/policyd-spf while reading input attribute name
 Apr 3 11:23:00 mail01 postfix/spawn[7398]: warning: command /usr/bin/python exit status 1
 Apr 3 11:23:00 mail01 postfix/psint/smtpd[7394]: warning: premature end-of-input on private/policyd-spf while reading input attribute name
 Apr 3 11:23:00 mail01 postfix/psint/smtpd[7394]: warning: problem talking to server private/policyd-spf: Success
 Apr 3 11:23:00 mail01 postfix/psint/smtpd[7394]: NOQUEUE: reject: RCPT from mail-pf0-f195.google.com[209.85.192.195]: 451 4.3.5 <email address hidden>: Recipient address rejected: Server configuration problem; from=<email address hidden> to=<email address hidden> proto=ESMTP helo=<mail-pf0-f195.google.com>

Here

 python -V
  Python 2.7.9

Changing in master.cf to

  /usr/bin/python3

where

 python3 -V
  Python 3.4.1

causes the same error for BOTH the =SPF & =AR Header_Types

So, the only case that seems to work is Python2 + Header_Type=SPF

[Scott Kitterman (kitterman)]

What operating system (distribution/release) are you using? Do you have the authres module installed?

[jasonsuser (b3411936)]

> What operating system (distribution/release) are you using?

 grep NAME /etc/os-release
  NAME="openSUSE Leap"
  PRETTY_NAME="openSUSE Leap 42.1 (x86_64)"
  CPE_NAME="cpe:/o:opensuse:opensuse:42.1"

 uname -rm
  4.5.0-7.g7f10552-default x86_64

> Do you have the authres module installed?

Yep

 pip2 -V
  pip 8.1.0 from /usr/lib/python2.7/site-packages (python 2.7)

 pip2 install authres
  Collecting authres
    Using cached authres-0.800.tar.gz
  Installing collected packages: authres
    Running setup.py install for authres ... done
  Successfully installed authres-0.800
 pip2 show authres
  ---
  Metadata-Version: 1.1
  Name: authres
  Version: 0.800
  Summary: authres - Authentication Results Header Module
  Home-page: https://launchpad.net/authentication-results-python
  Author: Julian Mehnle, Scott Kitterman
  Author-email: <email address hidden>
  License: Apache 2.0
  Location: /usr/lib/python2.7/site-packages
  Requires:
  Classifiers:
    Development Status :: 5 - Production/Stable
    Environment :: No Input/Output (Daemon)
    Intended Audience :: Developers
    License :: OSI Approved :: Apache Software License
    Natural Language :: English
    Operating System :: OS Independent
    Programming Language :: Python
    Programming Language :: Python :: 3
    Topic :: Communications :: Email :: Mail Transport Agents
    Topic :: Communications :: Email :: Filters
    Topic :: Software Development :: Libraries :: Python Modules

[Scott Kitterman (kitterman)]

Are you running it using python3 or python? The authres installation you're pointing to is for python2.7. If you're running it with python3, then you'll need authres installed in the proper location for a python3 module. Both the policy server and authres should work fine with either python2.7 or python3.4.

[Scott Kitterman (kitterman)]

Have a look at man policyd-spf in the "TESTING THE POLICY DAEMON" section. Using the same formulation as in your master.cf to call the policy server, test and see if it works from a command line. That will usually lead to a clearer error message.

[jasonsuser (b3411936)]

Are you running it using python3 or python?

This is Opensuse. It provides and can use /etc/alternatives.

/usr/bin/python == python2, or python3, depending on the alternatives switch.

In postfix, to avoid any such problems, II specify EITHER

/usr/bin/python2

or

/usr/bin/python3

I've both authres for python2 & python3 installed -- in the correct system locations for each.

The symptom's the same. Only the python2 + =SR case works right now.

I'll take a look at the "TESTING THE POLICY DAEMON" section ...

[jasonsuser (b3411936)]

Fyi,

pip3 -v install --upgrade authres pyspf
pip3 show authres pyspf
 ---
 Metadata-Version: 2.0
 Name: authres
 Version: 0.800
 Summary: authres - Authentication Results Header Module
 Home-page: https://launchpad.net/authentication-results-python
 Author: Julian Mehnle, Scott Kitterman
 Author-email: <email address hidden>
 Installer: pip
 License: Apache 2.0
 Location: /usr/lib/python3.4/site-packages
 Requires:
 Classifiers:
   Development Status :: 5 - Production/Stable
   Environment :: No Input/Output (Daemon)
   Intended Audience :: Developers
   License :: OSI Approved :: Apache Software License
   Natural Language :: English
   Operating System :: OS Independent
   Programming Language :: Python
   Programming Language :: Python :: 3
   Topic :: Communications :: Email :: Mail Transport Agents
   Topic :: Communications :: Email :: Filters
   Topic :: Software Development :: Libraries :: Python Modules
 ---
 Metadata-Version: 2.0
 Name: pyspf
 Version: 2.0.11
 Summary: SPF (Sender Policy Framework) implemented in Python.
 Home-page: http://pymilter.sourceforge.net/
 Author: Stuart D. Gathman
 Author-email: <email address hidden>
 Installer: pip
 License: Python Software Foundation License
 Location: /usr/lib/python3.4/site-packages
 Requires:
 Classifiers:
   Development Status :: 5 - Production/Stable
   Environment :: No Input/Output (Daemon)
   Intended Audience :: Developers
   License :: OSI Approved :: Python Software Foundation License
   Natural Language :: English
   Operating System :: OS Independent
   Programming Language :: Python
   Programming Language :: Python :: 3
   Topic :: Communications :: Email :: Mail Transport Agents
   Topic :: Communications :: Email :: Filters
   Topic :: Internet :: Name Service (DNS)
   Topic :: Software Development :: Libraries :: Python Modules

cd ./pypolicyd-spf-1.3.2

python3 setup.py build
 running build
 running build_py
 running build_scripts
 creating build/scripts-3.4
 copying and adjusting policyd-spf -> build/scripts-3.4
 changing mode of build/scripts-3.4/policyd-spf from 644 to 755
python3 setup.py install
 running install
 running build
 running build_py
 running build_scripts
 running install_lib
 copying build/lib/policydspfuser.py -> /usr/lib/python3.4/site-packages
 copying build/lib/policydspfsupp.py -> /usr/lib/python3.4/site-packages
 byte-compiling /usr/lib/python3.4/site-packages/policydspfuser.py to policydspfuser.cpython-34.pyc
 byte-compiling /usr/lib/python3.4/site-packages/policydspfsupp.py to policydspfsupp.cpython-34.pyc
 running install_scripts
 copying build/scripts-3.4/policyd-spf -> /usr/bin
 changing mode of /usr/bin/policyd-spf to 755
 running install_data
 creating /etc/python-policyd-spf
 copying policyd-spf.conf -> /etc/python-policyd-spf
 running install_egg_info
 Writing /usr/lib/python3.4/site-packages/pypolicyd_spf-1.3.2-py3.4.egg-info

find /usr/lib/python* -type f -name "*spf*"
 /usr/lib/python2.7/site-packages/spf.pyc
 /usr/lib/python2.7/site-packages/spf.py
 /usr/lib/python3.4/site-packages/policydspfsupp.py
 /usr/lib/python3.4/site-packages/policydspfuser.py
 /usr/lib/python3.4/site-packages/pypolicyd_spf-1.3.2-py3.4.egg-info
 /usr/lib/python3...

[jasonsuser (b3411936)]

If I repeat the same, but using pip2 & python2,

python2 /usr/bin/policyd-spf
    (empty)

[jasonsuser (b3411936)]

I removed all pip-installed python modules.

I removed the DIY'd pypolicyd-spf..

I installed distro-pkg for python-policyd-spf,

    python-policyd-spf-1.3.2-3.1.noarch

I exec using python2

with Header_Type=SPF, I get the "Received-SPF ..." in received-mail.

with Header_Type=AR, I get NO spf-related header or data in received-mail.

[Scott Kitterman (kitterman)]

On Wednesday, April 06, 2016 02:19:47 PM you wrote:
> python3 /usr/bin/policyd-spf
> Traceback (most recent call last):
> File "/usr/bin/policyd-spf", line 35, in <module>
> import spf
> File "/usr/lib/python3.4/site-packages/spf.py", line 110, in
> <module> import DNS # http://pydns.sourceforge.net
> ImportError: No module named 'DNS'

At least here you were missing py3dns (the pydns port to python3).

[Scott Kitterman (kitterman)]

The supplied config file should include a line like:

debugLevel = 1

Please change it to:

debugLevel = 5

Then run through the process described for testing the policy server in the man page.

You ought to then have a bunch of information in your mail log (in Debian it'd be /var/log/mail.log - it's been a decade since I touched opensuse, so I've no idea which log file it'd land in there).

Please copy/past all the log information into the bug. For what it's worth, AR support is, I'm confident, not really broken we just need to figure out what's missing or misconfigured.

[jasonsuser (b3411936)]

I switched to the distro-pkgs case, execing with python2.

I switched to debugLevel = 5

On send/receive of a testmail from gmail,

(1) with Header_Type=SPF, the rcvd mail has this in its headers, what I expected

 Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=209.85.220.67; helo=mail-pa0-f67.google.com; <email address hidden>; <email address hidden>

Here's the log

tail mail.log | grep -i policyd-spf

 Apr 7 06:34:14 mail01 policyd-spf[17175]: Starting
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "request=smtpd_access_policy"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "protocol_state=RCPT"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "protocol_name=ESMTP"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "client_address=209.85.220.67"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "client_name=mail-pa0-f67.google.com"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "client_port=36656"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "reverse_client_name=mail-pa0-f67.google.com"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "helo_name=mail-pa0-f67.google.com"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "<email address hidden>"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Read line: "<email address hidden>"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: spfcheck: pyspf result: "['None', '', 'helo']"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: None; identity=helo; client-ip=209.85.220.67; helo=mail-pa0-f67.google.com; <email address hidden>; <email address hidden>
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Header type: SPF; Authres ID (for AR): None
 Apr 7 06:34:14 mail01 policyd-spf[17175]: spfcheck: pyspf result: "['Pass', 'sender SPF authorized', 'mailfrom']"
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Pass; identity=mailfrom; client-ip=209.85.220.67; helo=mail-pa0-f67.google.com; <email address hidden>; <email address hidden>
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Header type: SPF; Authres ID (for AR): None
 Apr 7 06:34:14 mail01 policyd-spf[17175]: Action: prepend: Text: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=209.85.220.67; helo=mail-pa0-f67.google.com; <email address hidden>; <email address hidden>

(1) with Header_Type=AR, the rcvd mail's got no trace of *spf* in the headers at all.

and the log

tail mail.log | grep -i policyd-spf

 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "request=smtpd_access_policy"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "protocol_state=RCPT"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "protocol_name=ESMTP"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "client_address=209.85.192.194"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "client_name=mail-pf0-f194.google.com"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "client_port=34984"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "reverse_client_name=mail-pf0-f194.google.com"
 Apr 7 06:23:24 mail01 policyd-spf[15736]: Read line: "helo_name=mail-pf0-f194.google.c...

[jasonsuser (b3411936)]

I rm'd the distro pkg-installs.

I fixed the missing py3dns dependency

 pip3 show authres pyspf py3dns
  ---
  Metadata-Version: 2.0
  Name: authres
  Version: 0.800
  Summary: authres - Authentication Results Header Module
  Home-page: https://launchpad.net/authentication-results-python
  Author: Julian Mehnle, Scott Kitterman
  Author-email: <email address hidden>
  Installer: pip
  License: Apache 2.0
  Location: /usr/lib/python3.4/site-packages
  Requires:
  Classifiers:
    Development Status :: 5 - Production/Stable
    Environment :: No Input/Output (Daemon)
    Intended Audience :: Developers
    License :: OSI Approved :: Apache Software License
    Natural Language :: English
    Operating System :: OS Independent
    Programming Language :: Python
    Programming Language :: Python :: 3
    Topic :: Communications :: Email :: Mail Transport Agents
    Topic :: Communications :: Email :: Filters
    Topic :: Software Development :: Libraries :: Python Modules
  ---
  Metadata-Version: 2.0
  Name: pyspf
  Version: 2.0.11
  Summary: SPF (Sender Policy Framework) implemented in Python.
  Home-page: http://pymilter.sourceforge.net/
  Author: Stuart D. Gathman
  Author-email: <email address hidden>
  Installer: pip
  License: Python Software Foundation License
  Location: /usr/lib/python3.4/site-packages
  Requires:
  Classifiers:
    Development Status :: 5 - Production/Stable
    Environment :: No Input/Output (Daemon)
    Intended Audience :: Developers
    License :: OSI Approved :: Python Software Foundation License
    Natural Language :: English
    Operating System :: OS Independent
    Programming Language :: Python
    Programming Language :: Python :: 3
    Topic :: Communications :: Email :: Mail Transport Agents
    Topic :: Communications :: Email :: Filters
    Topic :: Internet :: Name Service (DNS)
    Topic :: Software Development :: Libraries :: Python Modules
  ---
  Metadata-Version: 2.0
  Name: py3dns
  Version: 3.1.0
  Summary: Python 3 DNS library
  Home-page: https://launchpad.net/py3dns
  Author: Scott Kitterman
  Author-email: <email address hidden>
  Installer: pip
  License: Python License
  Location: /usr/lib/python3.4/site-packages
  Requires:
  Classifiers:
    Development Status :: 5 - Production/Stable
    Environment :: No Input/Output (Daemon)
    Intended Audience :: Developers
    License :: OSI Approved :: Python License (CNRI Python License)
    Natural Language :: English
    Operating System :: OS Independent
    Programming Language :: Python :: 3
    Topic :: Internet :: Name Service (DNS)
    Topic :: Software Development :: Libraries :: Python Modules

I built installed policyd-spf 1.3.2 from upstream, using python3

 find /usr/lib/python* -type f -name "*spf*"
  /usr/lib/python3.4/site-packages/policydspfsupp.py
  /usr/lib/python3.4/site-packages/policydspfuser.py
  /usr/lib/python3.4/site-packages/pypolicyd_spf-1.3.2-py3.4.egg-info
  /usr/lib/python3.4/site-packages/__pycache__/spf.cpython-34.pyc
  /usr/lib/python3.4/site-packages/__pycache__/policydspfuser.cpython-34.pyc
  /usr/lib/python3.4/site-packages/__pycache__/policydspfsupp.cpython-34.pyc
  /usr/lib/python3.4/site-packages/spf.py

I switched ...

[jasonsuser (b3411936)]

Bingo!

amavisd was clobbering the header

 how to prevent amavis from removing Authentication Results?
  https://lists.amavis.org/pipermail/amavis-users/2012-May/001527.html

Changing in python-policyd-spf/policyd-spf.conf

- Authserv_Id = mail01.example.com
+ Authserv_Id = spf.mail01.example.com

results in recv'd mail with the following header now

 Authentication-Results: spf.mail01.example.com; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=209.85.220.65; helo=mail-pa0-f65.google.com; <email address hidden>; <email address hidden>)

So, policyd-spd + python3 + Header_Type=AR now working great.

Thanks!