Make Launchpad Bug Emails DMARC Compliant to avoid Launchpad bug mail considered spam

[Richard]

ISSUE:
When a comment is made on a bug, that comment is sent out by Canonical servers using the poster's email address. This won't work for domains that have dns policy records for email such as spf dkim or dmarc.

IMPACT:
A. ALL notifications where the issuer's domain has a published policy will be treated as spam or rejected
B. Canoncial's outbound smtp ip will be spam rated
C. dmarc policy holders will get reports on recipient's spammed domains
D. It is a bit of a privacy issue to distributes recipients domain names.

This won't work in 2016.

RECOMMENDED FIX:
Either use some clever canonical.com email construct like <email address hidden>
or a <email address hidden> address

[William Grant (wgrant)]

Launchpad is completely compliant with domains that use SPF and DKIM, as the envelope sender is @canonical.com. It's only operators who unwisely use a restrictive DMARC policy for user domains (breaking mailing lists, among other things) that are a problem.

[Richard (ismail-a)]

Here are the dmarc recommended options:

https://dmarc.org/wiki/FAQ#I_operate_a_mailing_list_and_I_want_to_interoperate_with_DMARC.2C_what_should_I_do.3F

Any "good" option will require modifying the from field, signing using dkim, supporting spf and provide a publicly verifiable certificate when sending.

It seems canonical does not do any of the four.

[Richard (ismail-a)]

current mail headers:
Reply-To: Bug 1589693 <email address hidden>
Received: from indium.canonical.com (indium.canonical.com [91.189.90.7])

Trouble 1: no SPF record
Trouble 2: no DKIM signature
Trouble 3: use of outdated TLS 1.0, should be 1.1 or 1.2
Trouble 4: MTA does not present a globally trusted X.509 certificate for its dns name

[Felix Schwarz (felix-schwarz)]

William Grant (wgrant) wrote on 2016-06-06:
> It's only operators who unwisely use a restrictive DMARC policy for user domains
> (breaking mailing lists, among other things) that are a problem.

Well, I think that ship has sailed. Big providers are employing strict DMARC policy and much more providers are willing to enforce these.

IMHO we just have to accept that email is not "anyone can send anything" anymore.

I just found a few launchpad messages in my spam folder. Besides Canonical having a DMARC record but no DKIM signature/SPF record the main culprit seems to be the From header which contains a commenter's email address.

Well that goes directly against DMARC and I can somehow understand their reasoning even if it is a bit annoying at times. Still I hope launchpad could be reconfigured to use an email address "@canonical.com" or "@bugs.launchpad.net".

[Thomas Ward (teward)]

Forgive me for a very blunt statement, but in reply to wgrant's statement about LP being compliant, it actually *isn't* being compliant. Simply altering "Sender" field and then implementing no changes to the "From" field is going to trigger any and all DMARC rules that have either a QUARANTINE or REJECT policy in place.

In having to handle DMARC compliance on a listserv implementation, and doing some heavier-duty testing with my DMARC compliant mail server(s), I discovered that not only does the Sender address and the outer Envelope have to not be from the DMARC-compliant addresses, the From field in headers is *also* verified, and can (and usually will) result in a failure case for all DMARC policies which restrict mail.

A large number of companies and groups implement strict DMARC handling, of which Launchpad will be noncompliant for in the future.

A prime example of this is a message sent by Launchpad today to the user 'nacc' which addressed the email as from me, because of my reply to a merge proposal made by them on something, and my critique on how they could fix an issue in that proposal. The relevant header bits are here:

Authentication-Results: mx.google.com;
       spf=pass (google.com: best guess record for domain of bouncesATcanonical.com designates 91.189.90.7 as permitted sender) smtp.mailfrom=bouncesATcanonical.com;
       dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=QUARANTINE) header.from=trekweb.org
...
From: Thomas Ward <email address hidden>
Sender: bouncesATcanonical.com

DMARC only cares about SPF and DKIM; it does not specifically care about whether messages were handled with TLS or not. Specifically, it wants the SPF of the sender and the DKIM of the "From" address to both validate properly. SPF passes because the Sender SPF records match properly with Canonical's SPF records. DMARC however fails because the domain trekweb.org publishes a DKIM record, and the DKIM signature does not match.

In *most* ListServ implementations that are DMARC compliant, there sits a mechanism to replace the "From" address with the address of the ListServ itself (sometimes with the raw mail header of "From: John Smith <listserv_AT_domain.com>", and sometimes a Reply-To of either the actual originator. The "From" field, therefore, validates against the domain of the listserv / system itself and *not* the actual originator. This solves the DMARC Compliance issue, as DMARC then checks the domain of the system itself (in this case, it would be canonical.com) and that will not fail.

Perhaps a setting could be added to user settings that would be "mask email address in notifications", which would then under the hood set the From header to be something like "User Name <bouncesATcanonical.com>" instead of the original from address?

[Starbeamrainbowlabs (sbrl)]

Just stumbled across this today. The core of the issue is that SPF and DKIM restrict who can send an email in the name of an email server - as I understand it DMARC is primarily a reporting system. To that end, it's not just DMARC that's the issue - SPF and DKIM are already causing issues too.

An email server's reputation is absolutely key to being able to deliver mail, and to that end most providers will implement fairly strict SPF and DKIM policies (to say nothing of DMARC) in order to protect their reputation.

I'd recommend rewriting to something like <email address hidden> or something.

[Jonathan Kamens (jik)]

I just started running opendmarc on my mail server, and it informed me this evening that an email message received from LaunchPad failed DMARC. The message came from `<email address hidden>`, so it's Canonical's own domain that is failing DMARC in this case, because although the domain has a DMARC record, it doesn't have an SPF record and launchPad doesn't do DKIM signatures.

Regarding the other issue discussed above, i.e., LaunchPad putting commenters' email addresses in the From: line of the message, this is clearly wrong given the status quo of email security. While I suppose someone could have reasonably claimed otherwise two and a half years ago, that ship has sailed. You can't be generating email messages that violate other domains' DMARC policies, period.

Both of these issues really need to be fixed.

[William Grant (wgrant)]

On 28/09/18 09:24, Jonathan Kamens wrote:
> I just started running opendmarc on my mail server, and it informed me
> this evening that an email message received from LaunchPad failed DMARC.
> The message came from `<email address hidden>`, so it's
> Canonical's own domain that is failing DMARC in this case, because
> although the domain has a DMARC record, it doesn't have an SPF record
> and launchPad doesn't do DKIM signatures.

Your DMARC configuration seems to be buggy, since the DMARC record for
canonical.com says no action should be taken -- it's just for testing.

_dmarc.canonical.com. 600 IN TXT "v=DMARC1; p=none; pct=100;
rua=mailto:<email address hidden>"

[Jonathan Kamens (jik)]

>Your DMARC configuration seems to be buggy, since the DMARC record for canonical.com says no action should be taken -- it's just for testing.

My DMARC configuration is not buggy. I didn't say that my mail server rejected the message, I said that it notified me that the message failed DMARC, which it did. The fact that the domain's DMARC policy says that no action should be taken about failing messages is orthogonal to the fact that the message in question failed DMARC.

I'm commenting just now about this because I apparently never got the notification about the comment above. I assume that's because it was dropped due to DMARC, although I can't say for sure since I only have email logs back to October 7.

However, I can say for certain that I'm missing other notifications from launchpad due to DMARC. My logs show that a few days ago I Launchpad emailed me about a comment added to a bug by a user with a yahoo.it email address. Launchpad put his email address in the From: line as outlined here in this bug, and yahoo.it has a p=reject DMARC policy, so my mail server rejected it.

Not, cool, Canonical. Please fix this.

[Starbeamrainbowlabs (sbrl)]

This is akin to impersonation. To that end, there's absolutely no reason for this to continue. Is Launchpad's code open-source?

[Colin Watson (cjwatson)]

The messages in question represent actions taken by the user whom Launchpad then lists in the From: line, and in most cases consist principally of body text that that user wrote. This is only "impersonation" if you take a very narrow MTA-centric view of who the person is. Morally, the situation is more like that of mailing lists resending messages they receive from users (with the main differences being that the origin of the message might be a mail user agent or might be an HTTP client, and that in some situations Launchpad will generate body text representing the user's actions rather than solely passing on lightly-modified body text provided by the user). If you're subscribed to this bug and you receive this message from me, it really is in essence a message from me and not from Launchpad.

Now, I'm not in general opposed to changes that adjust Launchpad's email headers so that it doesn't trip over DMARC implementations, although it would need quite a bit of care to go through all the relevant call sites and make sure that any such changes preserve the proper natural reply behaviour, and we should consider things like whether we want a From: address per (e.g.) bug or per Launchpad user. All this would take a fair amount of effort and discussion.

We may indeed nowadays be in an environment where we have little choice; but let's be clear that it would be a change made for operational reasons to improve the reliability of our mail delivery, and not for essentially moral reasons such as impersonation.

As for Launchpad being open source, yes it is, and there's a link with details at the bottom of every page on launchpad.net.

[Jonathan Kamens (jik)]

>We may indeed nowadays be in an environment where we have little choice;

I don't think there is any doubt about that, and anyone who continues to express doubt is either ignorant or naive about the current state of email.

I just missed another notification from Launchpad this morning because the comment I was being notified about was posted by a user at a domain with a DMARC reject policy.

Some notifications from Launchpad are not being delivered successfully because of how Launchpad is sending them. This isn't going to change; in fact, it's going to get worse over time. Plugging one's ears and loudly singing "la la la I'm not listening" is not going to change these facts.

Frankly, I can hate on DMARC as much as anybody, but I don't think pissing into the wind is a good long-term strategy.

>but let's be clear that it would be a change made for operational reasons to improve the reliability of our mail delivery, and not for essentially moral reasons such as impersonation.

100% agreed.

[Andrew Johnson (anj)]

A colleague who uses gmail for his MUA has been complaining for some time that he never gets Launchpad notification emails for comments that I make on his bug reports and merge requests, but he does get them from other team members. It's now clear to me having found this bug report that this Launchpad behavior is the reason for that problem. My employer publishes a strict DMARC policy (p=reject), whereas the other people he interacts with through Launchpad have either a lenient DMARC policy (p=none) or don't configure DMARC at all.

[Andrew Johnson (anj)]

I just found a similar and much older bug #31586 which doesn't mention DMARC until comment 33, but it is describing the same problem. This bug might therefore be a duplicate of that one, but this one has more specific details about DMARC which should be raising the urgency of a fix IMHO.

[Scott Kitterman (kitterman)]

There is an upcoming bolt on protocol for DMARC, called ARC:

https://tools.ietf.org/html/draft-ietf-dmarc-arc-protocol-23

This is the best shot at solving the problem with DMARC and mailing lists and as I understand it, the major mail providers have implemented or are planning to implement it. It's post IETF last call and in the RFC editor queue for publication.

Up to now, I think a reasonable case for claiming similarity between LP and a mailing list could be made. In an ARC world though I think that's less the case because the preferred mode of operations requires an inbound DKIM signature which, by definition, LP can't provide.

Operationally, this is only going to to get harder the longer a change isn't made. It' probably not relevant, but dkimpy (which last I knew LP used for DKIM verification) also supports ARC in recent versions.

[Scott Kitterman (kitterman)]

ARC is probably relevant for LP mailing lists as it gives a way to avoid from munging.

[Jonathan Kamens (jik)]

This bug isn't about mailing list hosted on Launchpad, it's about notifications generated by Launchpad. ARC has nothing to do with solving that problem.

[Andy Brody (abrody)]

Launchpad is really completely in the wrong here as far as DMARC compliance for its notification email is concerned.

The same rules apply to Facebook, GitHub, and any other site on the Internet. If the user's domain doesn't list you as an authorized sender, you shouldn't be impersonating them with the "From:" address.

Launchpad isn't somehow special here. Launchpad's email notices are indistinguishable from malicious forgery.

I was also very unpleasantly surprised that in addition to violating my domain's DMARC policy, Launchpad is disclosing my private email address despite my having checked the "Hide my email addresses from other Launchpad users" option.

[Jonathan Kamens (jik)]

I just missed an important notification from Launchpad about a bug I filed because the person who made the comment which generated the notification has an email address in a domain which enforces DMARC.

[Jonathan Kamens (jik)]

I just missed another notification because the person who made the comment which generated the notification has an email address in a domain which enforces DMARC.

It's unbelievable that this still isn't fixed over three years after the maintainers of Launchpad became aware of the issue.

[Colin Watson (cjwatson)]

We've been very short-staffed for a while, but are in the process of
growing our team again. This is too big a job to fit into spare cycles,
so I've nominated this for our roadmap for the next cycle.

[mdavidsaver (mdavidsaver)]

Another ping from someone effected by this situation.

[Dan Kortschak (dan-kortschak)]

Launchpad is just plain wrong here. Sticking its head in the sand is not going to get this solved.

[Colin Watson (cjwatson)]

Dan, we're not sticking our heads in the sand here, we just haven't managed to fit in the work yet. Telling us off isn't going to make it happen faster though!

[Dan Kortschak (dan-kortschak)]

Sorry, Colin.

The frustration comes from seeing that this issue has existed since 2016 and has seen significant push back on the actual validity of the concern. This is compounded by an observation that many issues filed at launchpad die by virtue of the hardware that they impact going out of service life before the issue is even triaged completely.

[Jonathan Kamens (jik)]

It's ironic that I got notified about Colin's response to Dan, but not Dan's original comment... because Dan's DMARC policy prevented me from seeing the notification about his comment.

[Tom Wardill (twom)]

The fix for this is now deployed, thanks to teward for the implenentation work.
Details here: http://blog.launchpad.net/notifications/bug-emails-now-use-the-bugs-address-in-the-from-header

If you experience any strange behaviour in email delivery, let us know.

[Colin Watson (cjwatson)]

Incidentally, we're aware that there are no DKIM signatures on our outgoing email yet, but that's something that Canonical's sysadmins will need to do and it's out of the hands of Launchpad developers. Bug 387321 tracks this.

[Brandon Applegate (vom)]

I too just noticed an email come in today (From: domain canonical.com) that failed DMARC on my server. I'm scratching my head as to why there is a DMARC record for canonical.com but no SPF or DKIM. DMARC is of course SPF OR DKIM (and that's an oversimplification as well, but...). So publishing a DMARC record without no SPF and no DKIM doesn't make any sense to me - it's guaranteed to fail every time (and yes I see there is a p=none in the policy which is why I accepted the mail...).

I also see over in the other bug:

https://bugs.launchpad.net/launchpad/+bug/387321

There is a comment from 2012 saying that SPF records won't be added...

Plenty of folks in the comments above already echoing these concerns, so I will try to not repeat them. I'm certainly sympathetic to lack of resources/time to properly fix (and test) things like this. But it seems like a much more serious and closer look needs to be had at the email policies and configurations.

PS: Looks like ubuntu.com and launchpad.net (as well as bugs.launchpad.net) don't have any SPF either. The difference is they don't publish DMARC, so there's no contradiction (this only seems to be the case for canonical.com).

[Gary Gapinski (5wtq-gary)]

On 6/4/20 8:34 PM, Brandon Applegate wrote:
> I too just noticed an email come in today (From: domain canonical.com)
> that failed DMARC on my server. I'm scratching my head as to why there
> is a DMARC record for canonical.com but no SPF or DKIM. DMARC is of
> course SPF OR DKIM (and that's an oversimplification as well, but...).
> So publishing a DMARC record without no SPF and no DKIM doesn't make any
> sense to me - it's guaranteed to fail every time (and yes I see there is
> a p=none in the policy which is why I accepted the mail...).

After staring at RFC7489 and particularly
https://tools.ietf.org/html/rfc7489#section-6.6.2, it seems that one
cannot obtain a fail in the absence of both an SPF declaration and a
DKIM-signed message. However, I will defer to any superior alien
intelligence which can provide a bible to thump regarding "absence of
pass is fail" as opposed to "lack of both SPF and DKIM precludes
pass/fail evaluation".

[Scott Kitterman (kitterman)]

On Thursday, June 4, 2020 10:15:45 PM EDT you wrote:
> On 6/4/20 8:34 PM, Brandon Applegate wrote:
> > I too just noticed an email come in today (From: domain canonical.com)
> > that failed DMARC on my server. I'm scratching my head as to why there
> > is a DMARC record for canonical.com but no SPF or DKIM. DMARC is of
> > course SPF OR DKIM (and that's an oversimplification as well, but...).
> > So publishing a DMARC record without no SPF and no DKIM doesn't make any
> > sense to me - it's guaranteed to fail every time (and yes I see there is
> > a p=none in the policy which is why I accepted the mail...).
>
> After staring at RFC7489 and particularly
> https://tools.ietf.org/html/rfc7489#section-6.6.2, it seems that one
> cannot obtain a fail in the absence of both an SPF declaration and a
> DKIM-signed message. However, I will defer to any superior alien
> intelligence which can provide a bible to thump regarding "absence of
> pass is fail" as opposed to "lack of both SPF and DKIM precludes
> pass/fail evaluation".

From that section:

> DMARC evaluation can only yield a "pass" result after one of the
> underlying authentication mechanisms passes for an aligned
> identifier.

So if a domain does neither SPF nor DKIM, there can't be a pass. DMARC only
has pass and fail.

Regardless, it is not uncommon for domains to publish DMARC records with
p=none in order to collect feedback. Canonical's DMARC record does that.

Scott K