InternalHosts evaluation fail due to error in UTF-8 string decoding

[Casper Bruun]

Summary

Evaluation of InternalHosts fail quietly due to error in utf-8 string decoding in Python 3.

Hence when sending mail from localhost (fx using Postfix sendmail or mail program) evaluation of InternalHosts always result in connection being deemed EXTERNAL (ie. self.internal_connection is False)

Debug output:

    "connect from localhost at ('127.0.0.1', 0) EXTERNAL"

Details

I could not get dkimpy-milter to sign mail generated on localhost.
After digging into the matter I found that the evaluation of InternalHosts in function _readConfigFile in config.py failed quietly:

    try:
        configData['AuthservID'] = _make_authserv_id(configData.get('AuthservID', 'HOSTNAME'))
        configData['IntHosts'] = HostsDataset(configData['InternalHosts'])
    except:
        pass

Adding debug output shows the reason for the failure is a TypeError:

   try:
        configData['IntHosts'] = HostsDataset(configData['InternalHosts'])
    except Exception as e:
        syslog.syslog("Could not make HostDataset from InternalHosts: {}".format(e))
        pass

Debug output:

   "Could not make HostDataset from InternalHosts: decoding str is not supported"

This originates from the function DataSetItemin class HostDataset in config.py.
In several places UTF-8 conversion is attempted using:

   str(self.item, "utf-8")

For example here:

   self.item = ipaddress.ip_address(str(self.item, "utf-8"))

In Python3 this is only working for bytes type strings (b'') resulting in the TypeError if the string is of type str.
Hence resulting in the TypeError "decoding str is not supported"

Suggested solution:

I am not sure situations where the strings that are being decoded could be either str or byte type strings could occur.
But since I encounter strings being python3 unicode str objects and the codebase seems to expect byte type strings (b''), I believe the safest option is to try for both.

For example:

    try:
       self.item = ipaddress.ip_address(str(self.item, "utf-8"))
    except TypeError:
       self.item = ipaddress.ip_address(self.item)

Environment

Ubuntu Linux OS v20.04.2
Python v3.8.10
postfix v3.4.13
dkimpy-milter v1.2.1-1

[Scott Kitterman (kitterman)]

Great bug report. Thanks.