Comment 1 for bug 1212214

Use of open("id_rsa.pub", 'rb') fix the issue.

But it seems to be little confusing, since my id_rsa.pub is in plaintext and not in binary format.
Moreover this issue is only Python3 specific and Python2 works well even without 'rb' mode specified.