DKIM validation fails with Python type error

[Colin Cogle]

DKIM validation seems to fail consistently with the newest version of DKIMpy-Milter. In the logs, the message "check_dkim: endswith first arg must be bytes or a tuple of bytes, not str" is logged. I'm not sure how to troubleshoot this further.

The email message header:
````
Authentication-Results: myservername.com; dkim=fail (endswith first
   arg must be bytes or a tuple of bytes, not str) header.d=externaldomain.com
   <email address hidden> header.a=rsa-sha256
````

My logs:
````
Oct 16 01:20:36 myservername.com dkimpy-milter[18783]: dkimpy-milter starting:18783 user:dkimpy-milter:postfix
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: connect from mx-out1.externaldomain.com at ('145.131.90.139', 57533) EXTERNAL
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: mail from: <email address hidden> ('SIZE=358',)
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: From: Sender <email address hidden>
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=externaldomain.com;
                                                                  s=2017-11; t=1571203254;
                                                                  bh=z85OKVJZHnmg3qFlSpLbpPCZ00irfBdrzQUtabiSl3A=;
                                                                  h=From:Subject:Date:To:From;
                                                                  b=p/+chVQfrOHDoiYfYiqfzR+n68yWVEPBPvbscYZisHfTMVu0VqZCdGpXM6aA5lSnU
                                                                   cJMmxfuwKgYiMBqT93tVrZjKlxWzKH64Ofchl5+tBsKa7SMGIdu46GiL5i2Pxy5Thz
                                                                   IoV2dzm20fpjI7O+PolF5LTt/RlBR1/QkEU/hdrbGQWI8QVHDr03UiZwP82AtpM5SD
                                                                   L8Zx9T+o/Bd5IENVW8iPSsNjAuFDfvZAIfZz3ZuNhEO/C1bLvc2LI+f9/sxgSS3Ueb
                                                                   XXj/Ecx83jC0iNhIcQOvvej64VoSwsTF5hOQEtTertPqdAcQtkAx86BZhSgNv+btyW
                                                                   7URoAIaEYmNBg==
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: content-type: text/plain;
                                                                  charset=us-ascii
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: content-transfer-encodeing: 7bit
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: REMOVE: myservername.com; spf=pass (mailfrom) smtp.mailfrom=externaldomain.com (client-ip=145.131.90.139; helo=mx-out1.externaldomain.com; <email address hidden>; receiver=<UNKNOWN>)
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: check_dkim: endswith first arg must be bytes or a tuple of bytes, not str
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: DKIM: Fail (saved as /tmp/tmpi9ffnemt.dkim)
Oct 16 01:20:55 myservername.com dkimpy-milter[18783]: Authentication-Results: myservername.com; dkim=fail (endswith first
                                                             arg must be bytes or a tuple of bytes, not str) header.d=externaldomain.com
                                                             header.a=rsa-sha256
````

[Scott Kitterman (kitterman)]

What you are seeing is an internal error being captured in the comment. I have a suspicion about the problem, but I haven't managed to replicate it. Would you please provide the real domain (d=) and selector (s=) for the signature. I need to look at the DKIM key record for the domain to test my theory.

[Scott Kitterman (kitterman)]

Even better, if you look at your logs, you can see a copy of the message was saved:

DKIM: Fail (saved as /tmp/tmpi9ffnemt.dkim)

If it's still there (or if you get another one), getting a complete, unaltered copy of the file (either attach it to the bug or email it to me as an attachment if you don't want it in the bug tracker) should make this pretty easy to figure out.

Thanks.

[Colin Cogle (signofzeta)]

I have hundreds of these failures. I'll email you a couple.

[Scott Kitterman (kitterman)]

Thanks. Unfortunately neither of them triggers an error here.

What operating system/distro is this on?
What version of dkimpy-milter are you running?
What version of dkimpy?
Do you have dnspython installed?
Do you have PyDNS installed?
Which python version are you using?

[Scott Kitterman (kitterman)]

If you are feeling adventurous and would like to get to the bottom of this quickly, here's something to try:

Find the dkimpy-milter module, probably some place like:

/usr/local/lib/python3.7/site-packages/dkimpy_milter
/usr/lib/python3.6/site-packages/dkimpy_milter
/usr/lib/python3/dist-packages/dkimpy_milter

If you've found the right place, it will have an __init__.py file in it. Make a copy of it for safety and then edit the file as follows:

You'll find two places that are very much like (they are at lines 302 and 314 in the current development version):

           except Exception as x:
                self.dkim_comment = str(x)
                if self.conf.get('Syslog'):
                    syslog.syslog("check_dkim: {0}".format(x))

Delete both of them and then restart the milter.

Once you do that, the next time this happens, the milter will die and you should get a full python traceback in your mail log. That should make it easy to figure out where to focus.

Only do this when you can keep an eye on it as once this happens, the milter will need to be manually restarted. Once this happens, copy back the backup you made and start the milter again.

I think it's perfectly reasonable if that's not something you're comfortable with, but I thought I'd ask since that would really help narrow this down more than anything.

[Colin Cogle (signofzeta)]

Thanks, I'll give that a try tonight. As for your previous question:

What operating system/distro is this on?
Ubuntu 18.04.3 LTS, kernel 4.15.0-1047-aws

What version of dkimpy-milter are you running?
1.2.0, cloned from the master branch.

What version of dkimpy?
0.9.3

Do you have dnspython installed?
No.

Do you have PyDNS installed?
No.

Which python version are you using?
I have 3.6.8 and 2.7.15+ installed.

[Scott Kitterman (kitterman)]

Thanks. You do need to have python3-dns or python3-dnspython installed. If you installed the packaged version of dkimpy (pthon3-dkim), it'll pull in python3-dnspython in automatically. I assume you have one of them, since no verification would work if you didn't.

[Colin Cogle (signofzeta)]

I tried installing python3-dns, but apt-get said it was already installed (version 3.1.1-1). I did install python3-dnspython, but there was no change in behavior. I'll try removing your error-checking code and seeing what it tells me.

[Colin Cogle (signofzeta)]

You can close this ticket. I tried what you said, but the line numbers didn't match up. Commenting out the except block didn't yield anything useful.

I tried removing all traces of dkimpy-milter from my system manually and through pip3, reinstalling from the Git clone, and restoring my keys and dkimpy-milter.conf file. Now everything is working perfectly. Sorry for the trouble!

[Scott Kitterman (kitterman)]

Thanks for letting me know.

I try to keep master in a reasonably functional state, but things do happen. It may also be that something was broken when you pulled from git last that I fixed in the meantime. A certain amount of breakage in master is inevitable, I suppose. I guess if something comes up again, update from git first and see if I've fixed it. usually the broken states are short since they most often occur when I'm in the middle of testing.