validation by yahoo/hotmail/dkimpy fails because of whitespace in h= or header folding issues

[Martin Pool]

Hello. I am new to DKIM, and might be doing something silly.

I build a message using email.mime, call as_string() on it, construct a DKIM object with the msg, and call sign. I prepend that signature onto my message, and send it. An email to a google apps account gives me "dkim=pass. An email to a yahoo account gives "dkim=permerror (bad sig)". This is using the same code, same from email and domain, etc.. I get the same results using pydkim-3 and with the newer dkimpy-5.1.

While I was still using pydkim-3, I tried different things, and managed to get both yahoo and gmail to give me a pass. I changed the sign function to omit the spaces around the colon in the "h=" list, and removed the call to the fold utility in the sign function so that the signature header was all on one long line. With those changes, yahoo and google were both happy.

Any suggestions?

Thanks,
Rob

[Martin Pool (mbp)]

Hi, Rob, thanks for reporting this.

In <http://tools.ietf.org/html/rfc4871#section-3.5> describing "h=":

  Folding whitespace (FWS) MAY be included on either side of the colon separator.

It looks like Yahoo doesn't handle that properly. Perhaps it is worth trying to report an issue to them.

I think a patch to pydkim that makes it not put whitespace in the h= field would be fine. You should add a comment explaining the issue with Yahoo. If there is no FWS I think it won't fold; otherwise we could use strict signing.

[Stuart Gathman (stuart-gathman)]

Possible patch for compatibility with yahoo braindamage. Passes test suite.

--- dkim/__init__.py 2012-02-03 22:10:49 +0000
+++ dkim/__init__.py 2012-02-23 03:06:30 +0000
@@ -238,7 +238,7 @@
             j = i + 1
         pre += header[:j] + b"\r\n "
         header = header[j:]
- return pre + header
+ return (pre + header).replace(' : ',':')

A better way to do it would be to have the fold method break on either ':' or ' '.

[Rob Nichols (0x2jjt-rob-0tlbog)]

See related question for a little discussion. It's the folding that seems to bother yahoo, and not the space. Here's a patch of what I mean. I've asked the yahoo feedback loop for input, and will report back if/when they reply.

--- dkim/__init__.py 2012-02-22 21:56:26.000000000 -0600
+++ dkim/__init__.py 2012-02-22 21:19:23.000000000 -0600
@@ -441,7 +441,8 @@
     # record what verify should extract
     self.include_headers = tuple(include_headers)

- sig_value = fold(b"; ".join(b"=".join(x) for x in sigfields))
+ sig_value = b"; ".join(b"=".join(x) for x in sigfields)
     dkim_header = (b'DKIM-Signature', b' ' + sig_value)
     h = hashlib.sha256()
     sig = dict(sigfields)

[Martin Pool (mbp)]

It would be good to know if folding is a problem only in strict mode
or only in relaxed mode or both.

[Rob Nichols (0x2jjt-rob-0tlbog)]

I'll check tomorrow and let you know.

[Rob Nichols (0x2jjt-rob-0tlbog)]

Previous tests done in the default, "simple" mode for both headers and body.

Here are the results using dkimpy-5.1 as-is, meaning is still has the space in "h=" and the header is folded.

(simple, simple) -> google pass / yahoo dkim=permerror (bad sig)
(relaxed, relaxed) -> google pass / yahoo pass
(relaxed, simple) -> google pass / yahoo pass
(simple, relaxed) -> google pass / yahoo dkim=permerror (bad sig)

No response yet from yahoo. For my own usage, I'm going with stock dkimpy-5.1, and (relaxed, simple) conanicalization.

[Rob Nichols (0x2jjt-rob-0tlbog)]

Okay, one more piece of information.

MSN/live/hotmail respond like yahoo. It fails with simple, but passes with relaxed. I have no idea why...?

Can you change the title of this bug? Or should I move the additional details to the question and open a new bug?

[Stuart Gathman (stuart-gathman)]

In simple mode, you are supposed to hash header fields without unfolding them, including the DKIM-Signature header field. BUT, you MUST treat the b= param of the current signature as if it were the empty string. That leaves an ambiguity in the spec. What if the b= value, which is long, is then folded? When replacing the b= value with empty string, should you also remove any FWS inside the value? Adjacent to the value? I can see different interpretations. dkimpy puts b= as the last sig param, and does not fold again after signing (thus the b= line can be longer than 72 chars). However, perhaps your mail relay is folding the headers? And google is actually being more tolerant than it should? Or the difference is folding in the b= param.

[Scott Kitterman (kitterman)]

I asked one of the original DKIM developers about this (who also maintains their own DKIM library) and he said:

Can you capture a copy of the canonicalized header and body of a message
that goes through that software? Seeing the original message and the raw
data that went through the hash will help.

I seem to remember RFC4871 had an erratum about this that I think we fixed
in RFC6376, but I'll have to go back and double-check later this evening.

So if someone could capture that, it would be helpful.

[Scott Kitterman (kitterman)]

This erratum for RFC 4871 looks like it covers this exact issue:

http://www.rfc-editor.org/errata_search.php?eid=1596

[Stuart Gathman (stuart-gathman)]

Yep. That certainly sounds like this. And you can work around it by ensuring there is no whitespace or folding before or after the b= value. And dkimpy already does this by ensuring the b= is last, and folding before computing sig and appending value. I suspect that the header is getting refolded later on by some other software (which is calling dkimpy) - which could trigger this ambiguity.

[Rob Nichols (0x2jjt-rob-0tlbog)]

Yes, I can see that this might be related, but I don't think this is quite everything. When I removed the call to fold() in the DKIM.sign(), which made the signature one long line, yahoo accepted the signature. (However, MSN still did not.) What I've decided to do with my current project is use (relaxed, simple) canonicalization. That made google, yahoo, and MSN happy. It avoids any related confusion about this errata.

What about changing the default c14n to (relaxed, simple)? I think it would move closer to "just working" for new users of the module.

[Scott Kitterman (kitterman)]

I'd prefer a proper resolution of the issue, but we may have to do that. I'm in communication with the author of the DKIM library used by Yahoo.

[Scott Kitterman (kitterman)]

Would you please run your tests with the different canonicalization methods again sending to <email address hidden> and report your results. It runs the same DKIM library as Yahoo! and will return the body of your messages as an attachment so you can see if it's modified in transit.

[Rob Nichols (0x2jjt-rob-0tlbog)]

Certainly, happy to. (Looking at my schedule, it will probably be a few days...)

[Scott Kitterman (kitterman)]

ping about getting signed mails that fail at yahoo/hotmail so I can get them to look into it.

[Rob Nichols (0x2jjt-rob-0tlbog)]

Thanks for the reminder, and my apologies for the the delay. I will get to this as soon as I can, hopefully in the next couple of days.

On Thursday, April 5, 2012 at 8:12 PM, Scott Kitterman wrote:

> ping about getting signed mails that fail at yahoo/hotmail so I can get
> them to look into it.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/939128
>
> Title:
> validation by yahoo/hotmail fails because of whitespace in h= or
> header folding issues
>
> Status in Module for DKIM signing and verification in Python:
> Confirmed
>
> Bug description:
> Hello. I am new to DKIM, and might be doing something silly.
>
> I build a message using email.mime, call as_string() on it, construct
> a DKIM object with the msg, and call sign. I prepend that signature
> onto my message, and send it. An email to a google apps account gives
> me "dkim=pass. An email to a yahoo account gives "dkim=permerror (bad
> sig)". This is using the same code, same from email and domain, etc..
> I get the same results using pydkim-3 and with the newer dkimpy-5.1.
>
> While I was still using pydkim-3, I tried different things, and
> managed to get both yahoo and gmail to give me a pass. I changed the
> sign function to omit the spaces around the colon in the "h=" list,
> and removed the call to the fold utility in the sign function so that
> the signature header was all on one long line. With those changes,
> yahoo and google were both happy.
>
> Any suggestions?
>
> Thanks,
> Rob
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/dkimpy/+bug/939128/+subscriptions
>
>

[Scott Kitterman (kitterman)]

I think for Ubuntu precise (12.04) I'm going for the relaxed/simple option as a workaround since we're very close to release.

[Scott Kitterman (kitterman)]

dkimpy (0.5.1-1ubuntu1) precise; urgency=low

  * Change default header signing canonicalization from 'simple' to
    'relaxed' to avoid interoperability problems like those described in
    LP 939128
    - This is a work-around, not a fix

 -- Scott Kitterman <email address hidden> Wed, 18 Apr 2012 01:00:18 -0400

[Stuart Gathman (stuart-gathman)]

Changed defaults in revision 86

[Rob Nichols (0x2jjt-rob-0tlbog)]

Sorry for the delay. I no longer have access to some of the test accounts. I created new accounts, and verified that the results were the same as previously reported. I wrote each messages to a file. Looks like I can only attach one, so I'll go with msn (actually live.com). Can attach the others to additional posts if helpful. This is using dkimpy-5.1, and setting c14n to simple/simple. Results were:

google pass
yahoo dkim=permerror (bad sig)
live X-DKIM-Result: Fail(t)

I sent a copy to <email address hidden>, and will post the response when I get it.

[Scott Kitterman (kitterman)]

Additional test data.

Forwarded Message ----------

Subject: Re: dkimpy interop issue with Microsoft
Date: Monday, April 23, 2012, 01:52:42 PM
From: Rob Nichols <email address hidden>
To: Scott Kitterman <email address hidden>

Very sorry for the slow response time. Here are test messages as sent to
yahoo, msn an gmail. Gmail passed, both yahoo and msn failed. Sign generated
with 5.1 using c14n simple/simple. Guess you can see simple/simple in the
header...

On Monday, March 12, 2012 at 5:10 AM, Rob Nichols wrote:

> Great! It will be a few more days before I'm back and can generate those
emails, but I'll get to it as soon as I can.
>
> On Wednesday, March 7, 2012 at 2:02 PM, Scott Kitterman wrote:
>
> > I'm writing you directly because I've got a contact at Microsoft that will
> > work with us on solving your issue. He's asked for signed messages that
fail.
> > If you can generate them and then attach them in an email to me I can pass
> > them on and they'll have an engineer look into it.
> >
> > Scott K
>

-----------------------------------------

[Stuart Gathman (stuart-gathman)]

Aha! Notice that for all 3 messages (attached to comment #22) the DKIM-Signature was folded on the email services, but *not* folded in the original as sent out (attached to comment #21). So as I suspected in comment #11, something else is refolding the DKIM-Signature header (probably an MTA, probably Rob Nichols MTA). According to the spec, this should not affect the signature, the DKIM-Signature being verified is treated specially, and folding whitespace is specifically allowed.

For what it's worth, dkimpy also fails to validate the message when b= is folded. :-( . I suggest we

1) double check our interpretation of the spec.

2) fix dkimpy to accept folding whitespace within the b= parm of the DKIM-Signature being validated.

3) *Then* loudly tell M$ and Yahoo they are doing it wrong. :-)

[Stuart Gathman (stuart-gathman)]

From the RFC:

   b= The signature data (base64; REQUIRED). Whitespace is ignored in
       this value and MUST be ignored when reassembling the original
       signature. In particular, the signing process can safely insert
       FWS in this value in arbitrary places to conform to line-length
       limits. See Signer Actions (Section 5) for how the signature is
       computed.

gmail gets it right. dkimpy, live.com, yahoo get it wrong.

[Rob Nichols (0x2jjt-rob-0tlbog)]

The file attached to #21 and #22 should be identical. I attached #21, and Scott attached #22 from files I emailed him. The files on #21 and #22 were captured before being sent. Neither is post mailing. I sent these with twisted, which I think does *not* fold lines... but I could be wrong.

I looked at my live.com test address. It has a "view message source" option, that opens a new window and shows the message with minimal formatting. I've never much trusted such pages to show the "real" source... Anyway, when I compared the DKIM header in what I sent to what live.com showed, they match. They have the same folding. I hesitate to draw too much from that, but it would seem to mean that no additional folding happened between signing and delivery.

[Stuart Gathman (stuart-gathman)]

Added test case with revision 87.

[Stuart Gathman (stuart-gathman)]

Fix for dkim.verify commited with revision 88. This does not, of course, fix yahoo and live. So the default will still be relaxed header c14n.

[Rob Nichols (0x2jjt-rob-0tlbog)]

Just for completeness, thought I would attach what live.com returns via "view message source." This is corresponds to the file attached to #21 above. I realize this issue is closed, but thought having everything together might be useful to msn/yahoo...?

Thanks all!

[Scott Kitterman (kitterman)]

I just cherrypicked this for the Ubuntu release this Thursday, so it should be
good there.