SPF rejects mail on unsupported SPF RR type

[Steven McCoy]

Binary package hint: libmail-spf-query-perl

Not all DNS servers support the new SPF RR type and reply unsupported or SERVFAIL to queries for it. libmail-spf-query-perl will cause mail to be 451 rejected in postfix-policyd-spf-perl.

Noted here that some email servers will incorrectly reject mail if unsupported:

http://www.simpledns.com/newsitem.aspx?id=2171

Example rejection:

Nov 9 15:26:11 hyolee postfix/policy-spf[15786]: : Policy action=DEFER_IF_PERMIT SPF-Result=smtp03.hkpc.org: 'SERVFAIL' error on DNS 'SPF' lookup of 'smtp03.hkpc.org'

A possible workaround is editing /usr/share/perl5/Mail/SPF/Server.pm, line 440 to treat unsupported operation as not found:

    $packet->header->rcode =~ /^(NOERROR|NXDOMAIN|SERVFAIL)$/

This produces a SPF: none result:

Nov 9 16:52:45 hyolee postfix/policy-spf[29847]: : Policy action=PREPEND Received-SPF: none (hkpc.org: No applicable sender policy available) receiver=hyolee.miru.hk; identity=mfrom; <email address hidden>"; helo=smtp03.hkpc.org; client-ip=202.130.102.27

Something else is happening because the email is still rejected due to a policyd timeout:

Nov 9 16:53:54 hyolee postfix/smtpd[28617]: warning: problem talking to server private/policy: Connection timed out
Nov 9 16:53:54 hyolee postfix/smtpd[28617]: NOQUEUE: reject: RCPT from smtp03.hkpc.org[202.130.102.27]: 451 4.3.5 Server configuration problem; from=<email address hidden> to=<email address hidden> proto=ESMTP helo=<smtp03.hkpc.org>

If I comment out the entire SPF lookup section:

    # Query for SPF type RRs first:
# try {
# my $packet = $self->dns_lookup($domain, 'SPF');
# push(
# @records,
# $self->get_acceptable_records_from_packet(
# $packet, 'SPF', \@versions, $scope, $domain)
# );
# }
# catch Mail::SPF::EDNSError with {
# push(@dns_errors, shift);
# };

That sending host now works:

Nov 9 17:41:36 hyolee postfix/policy-spf[1954]: handler sender_policy_framework: is decisive.
Nov 9 17:41:36 hyolee postfix/policy-spf[1954]: : Policy action=PREPEND Received-SPF: none (hkpc.org: No applicable sender policy available) receiver=hyolee.miru.hk; identity=mfrom; <email address hidden>"; helo=smtp03.hkpc.org; client-ip=202.130.102.27
Nov 9 17:41:36 hyolee postfix/smtpd[1835]: NOQUEUE: filter: RCPT from smtp03.hkpc.org[202.130.102.27]: <email address hidden>: Sender address triggers FILTER smtp-amavis:[127.0.0.1]:10024; from=<email address hidden> to=<email address hidden> proto=ESMTP helo=<smtp03.hkpc.org>
...

[Scott Kitterman (kitterman)]

Which version of postfix-policyd-spf-perl are you using? I suspect this belongs to mail-spf-perl as all the packaged versions of the Postfix policy server use that library and not mail-spf-query-perl.

[Steven McCoy (fnjordy)]

I have 2.002-0ubuntu2 on one host, and a backported 2.004-0ubuntu2~feisty1 on another, both the same.

[Julian Mehnle (jmehnle)]

Hi, I am the upstream author of Mail::SPF. Two things first:

 1. The policyd timeout is indeed a postfix-policyd-spf-perl issue. Maybe I can add an option to Mail::SPF that allows a shorter timeout period (than the Net::DNS default) to be specified.
 2. The SERVFAIL DNS status code (RCODE 2) does NOT mean "unsupported operation". According to RFC 1035, it means "The name server was unable to process this query due to a problem with the name server". There is another status code, "Not Implemented" (RCODE 4), that means "The name server does not support the requested kind of query". Please do not conflate the two. Thus, interpreting SERVFAIL as "not found" is inappropriate because it could just as well mean "out of memory" or "out of database connections, try again later".

Now on to the Mail::SPF issue. If you look at <http://spf.pastebin.com/f3b588a5a>, the name servers responsible for smtp03.hkpc.org really seem to have a much bigger problem than not supporting the SPF RR type. Based on this, the TempError returned by Mail::SPF appears to me as being absolutely justified, regardless of Mail::SPF's handling of the SPF RR type.

However, concerning the handling of the SPF RR type, understand that while Mail::SPF queries for both the SPF and TXT RR types (in this order), only the SPF-type lookup failing isn't enough for TempError to be returned. If the SPF-type lookup fails but the (following) TXT-type lookup succeeds, Mail::SPF silently uses the TXT response to calculate an SPF result. Only if /both/ the SPF- and TXT-type lookups fail does it return an error (based on the result of the SPF-type lookup, which was the first performed).

[Scott Kitterman (kitterman)]

What does postconf smtpd_policy_service_timeout return on your servers?

That defaults to 100s and according to your logs, it's returning sooner than that. A totally braindead DNS server can suck up up to 80s (20s SPF HELO lookup + 20s TXT HELO lookup + 20s Mail From SPF lookup + 20s Mail From TXT lookup).

[Julian Mehnle (jmehnle)]

Right. The default timeout for a DNS query via UDP effectively is 20 seconds (4*5s as per Net::DNS::Resolver's retrans and retry properties).

[Scott Kitterman (kitterman)]

I've discussed this with the Mail::SPF developer (and Debian maintainer) and he declines to disable type SPF queries. I don't think Ubuntu should carry a permanent diff for this, so I'm won't fixing that aspect of the issue.

[Scott Kitterman (kitterman)]

80 seconds is a long time to wait for a final answer. A change to cut the DNS timeouts in half (so that 40 seconds is the maximum) has been committed to the upstream SVN repository. Once that's tested, it will get released, uploaded to Debian, and then sync'ed for Hardy.

[Scott Kitterman (kitterman)]

The new version is uploaded to Debian and sync has been requested. See Bug #176726 for status on the sync.

[Scott Kitterman (kitterman)]

2.005 drops the timeouts so that these kinds of problems should be less common. If you want to avoid Type SPF queries entirely you'll have to switch to the Python SPF policy server.

[Scott Kitterman (kitterman)]

The option to disable type SPF queries was added in mail-spf-perl 2.006.

[Scott Kitterman (kitterman)]

Reopening upstream now that we can fix this more properly.

[Scott Kitterman (kitterman)]

Fix committed to Trunk, rev 63. Will be in the next release.

[Scott Kitterman (kitterman)]

Released with version 2.008.