DMARC munging fails on subdomains that use parent domain policy

[Robert Mathews]

An address ending with "@reply.yahoo.com" posted a message to a list that has "dmarc_moderation_action" set to "Munge From".

This causes IsDMARCProhibited() in Utils.py to lookup TXT records for "_dmarc.reply.yahoo.com". It finds none, so Mailman does not apply the munging.

However, in this situation, DMARC "clients" apparently need to look "up the chain" at "_dmarc.yahoo.com". See RFC 7489 section 4.3 point 7, and section 6.6.3 point 3 ("Organizational Domain"), and here's an example of it in action:

 https://dmarcian.com/dmarc-inspector/reply.yahoo.com

The result of this bug is that mail from a subdomain like "@reply.yahoo.com" does not get munged, but does get rejected downstream by yahoo.com/gmail.com etc. for failing DMARC policy.

(I'm using Mailman 2.1.20.)

[Mark Sapiro (msapiro)]

Thank you for your report.

The fix I committed is something of a Kludge and will test more domains that just the From: domain and Organizational Domain as determining the Organizational Domain in general is a challenge, but hopefully it will produce the correct result in at least most cases.

[Jim Popovitch (jimpop)]

The real problem is Yahoo not putting a DMARC record at _dmarc.reply.yahoo.com.

1) It would be ridiculous to implement code, which would need to be updated with every new TLD, to properly determine a parent domain.

2) Checking the parent domain really only applies when the signing is in relaxed mode, certainly not strict.... so that needs to be checked to.

[Mark Sapiro (msapiro)]

RFC 7489 is pretty clear.

Sec 6.6.2 gives the first 2 steps as:

   1. Extract the RFC5322.From domain from the message (as above).

   2. Query the DNS for a DMARC policy record. Continue if one is
       found, or terminate DMARC evaluation otherwise. See
       Section 6.6.3 for details.

and 6.6.3 gives the first 3 steps as:

   1. Mail Receivers MUST query the DNS for a DMARC TXT record at the
       DNS domain matching the one found in the RFC5322.From domain in
       the message. A possibly empty set of records is returned.

   2. Records that do not start with a "v=" tag that identifies the
       current version of DMARC are discarded.

   3. If the set is now empty, the Mail Receiver MUST query the DNS for
       a DMARC TXT record at the DNS domain matching the Organizational
       Domain in place of the RFC5322.From domain in the message (if
       different). This record can contain policy to be asserted for
       subdomains of the Organizational Domain. A possibly empty set of
       records is returned.

I.e. If the From: domain doesn't have a valid DMARC policy record, you MUST query the Organizational Domain.

All of this occurs before any DKIM and SPF checks which are steps 3 and 4 of the sec 6.6.2 procedure.

I have refactored the fix to use the data from https://publicsuffix.org/list/public_suffix_list.dat and apply the algorithm described at https://publicsuffix.org/list/ to determine the organizational domain and to query that in cases where the From: domain has no DMARC record and the computed organizational domain is different.

The refactored fix will read the data only once after Mailman is (re)started and build a dictionary kept in global memory to use in implementing the algorithm. See http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/1620 for the original fix and http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/1621 for the refactoring.

[Jim Popovitch (jimpop)]

Nice stuff Mark.

The thing that I don't see is that we must determine the Org policy on how to handle subdomains. If we don't find a policy at _dmarc.reply.yahoo.com, then we still need to parse _dmarc.yahoo.com to see if it can even be applied to a subdomain ("adkim", "aspf", and "sp" ). It's messy stuff. I firmly believe people crafted DMARC for job security. :-)

[Mark Sapiro (msapiro)]

I think you are confusing two things: One is what DMARC policy if any to apply to a message and the other is how to determine whether a message passes DMARC.

In our case, we want only to determine what if any DMARC policy will be applied, and the RFC says it's the policy of the From: domain if the From: domain has one and if not and the From: domain is a subdomain of an organizational domain, the organizational domain's policy if any applies.

A recipient will then look at things like DKIM sigs and SPF and strict vs. relaxed alignment to determine if the message has a valid SPF or DKIM sig alligned with the From: domain, but we already know that we broke the incoming DKIM sig and it won't validate in any case or we wouldn't be applying DMARC mitigations. Thus, we need to know what if any DMARC policy will be applied in order to know whether to apply our mitigations.

[Jim Popovitch (jimpop)]

Let's look at this from a different angle. Say a message arrives (admittedly from a poorly configured setup) as From: <email address hidden> do you use the DMARC policy defined at _dmarc.digitalocean.com ?

[Mark Sapiro (msapiro)]

Following RFC7489 sec 6.6.3, we first look up the TXT records at _dmarc.vps1234.digitalocean.com. That returns no record with v= the current DMARC version (actually none at all) so we then query _dmarc.digitalocean.com (the organizational domain in this case) and find a valid v=DMARC1 p=reject record, so if we are Mailman doing this to determine whether or not to apply dmarc_moderation_action, we do apply the action.

If Mailman didn't apply dmarc_moderation_action and just processed the post, the ultimate receiver would get to the same point and determine the applicable DMARC policy is reject and then proceed with the DKIM and SPF checks to determine if the message passes DMARC.

If we assume that the message didn't go via a list and instead went directly to the ultimate receiver, that receiver would determine the DMARC policy was reject in the same way. Then it would check DKIM and SPF. This is the first point at which adkim or aspf would be checked for strict/relaxed alignment between the From: domain and the DKIM signing or SPF domain.

As Mailman we don't care about any of this because we aren't trying to validate DMARC on the incoming message, and we assume that DMARC won't validate against the outgoing message if we don't alter the From: domain because we aren't the From: domain for SPF and we transform the message in ways that break the DKIM sig, so all we want to know is what DMARC policy will be applied by the receiver.