Email with Upper Case causes Bug message

[Kathy Gee]

Seems like members with upper case characters in their subscribed email address causes a "bug" message when trying to access their options page. The "bug" message also appears when trying to access the member's option page via the Admin page. Below is the traceback:

admin(10092): Traceback (most recent call last):
admin(10092): File "/usr/local/mailman/scripts/driver", line 101, in run_main
admin(10092): main()
admin(10092): File "/usr/local/mailman/Mailman/Cgi/options.py", line 821, in main
admin(10092): options_page(mlist, doc, user, cpuser, userlang)
admin(10092): File "/usr/local/mailman/Mailman/Cgi/options.py", line 865, in options_page
admin(10092): subscriber_id = subscriber.getSubscriber_id_raw_or_die(user)
admin(10092): File "/usr/local/mailman/Mailman/DlistUtils.py", line 98, in getSubscriber_id_raw_or_die
admin(10092): raise ErrorsDlist.InternalError
admin(10092): InternalError

[Kathy Gee (kathyg)]

Response from Kanika on 5/19/10:

I tried to subscribe a user with One letter of his email address in capital (locally on my system) and I am getting same error.That is when I try to log in to the membership configuration page I get Mailman has hit a bug.I compared the lines of code with Old DlistUtils.py (before STORM implementation) and they were as follows:

def getSubscriber_id_raw(mlist, addr):
    if addr == None:
# syslog('info', 'getSubscriber_id_raw called with addr=None')
        return None
    return executeSQL(mlist,
                      ["SELECT subscriber_id \
                      FROM subscriber \
                      WHERE mailman_key = '%s'" % addr.lower()])

As you see 'addr.lower()' is being used since long time before.I wonder why no such bug hit since then.The storm implementation is the exact replica of the above lines.

I printed out the value of 'addr' on the info log and found that addr contains the email address all in lower case letters.If you look at the error log then it mentions about line 865 in Mailman/Cgi/option.py before mentioning about DlistUtils.py which is:

subscriber_id = subscriber.getSubscriber_id_raw_or_die(user)

The user here is again email address all in small letters.

So I made a minor change:

I changed the line:

 result = self.store.find(Subscriber,Subscriber.mailman_key == unicode(addr.lower(),"utf-8"))

to

result = self.store.find(Subscriber,Subscriber.mailman_key.lower() == unicode(addr,"utf-8"))

and then the bug was resolved for the moment.

But when I tried to change the membership option and hit 'submit my changes' button then again Mailman hit a bug.After reading the bug report I found that the line in DlistUtils.py that was giving the error was the one in red below:

"
def setDisable(self, member, flag):
        """Disable/enable delivery based on mm_cfg.DisableDelivery"""
        command = "result = self.store.find(Subscriber,Subscriber.mailman_key == unicode(member,'utf-8'))\noldval = [(subscriber.suppress) for subscriber in result]\n"
        if DEBUG_MODE:
            syslog('info', 'DlistUtils(setDisable):Executing query:\n%s\n Member whoes suppress value is to be found \n %s', command,member)
        result = self.store.find(Subscriber,Subscriber.mailman_key == unicode(member,"utf-8"))
        oldval = [(subscriber.suppress) for subscriber in result]
    if oldval == []:
        if DEBUG_MODE:
            syslog('info','oldval is an empty list.\nThis can happen either because of\n 1)Permission issues (Do a: bin/check_perms)\n 2)Inconsistency between database and pickle files (A user is in the database but not in pickle files or vice versa,Do a bin/find_problems.py)')"

Again on priniting the value of 'member' in the info log I found that member also gives the email address of a subscriber all in lowercase.

So shall we make changes in the code such that whenever the email address of a subscriber is added(or changed) in the database it is added in lowercase(As everwhere email is being used in lowercase) or shall I just give a call to lower() function wherever Susbcriber.mailman_key is called to be on a safer side?

...

[Robin J (robin-jeffries)]

OK, I think I get it.

The problem is that somehow (I'm not sure how, as it doesn't look like it is possible to create such addresses now), there are people in the database who have email addresses with upper case letters. So we need to assume that those exist -- I'm not at all sure how to eliminate them (though someone with more sql knowledge than I have might be able to). So I think that we need to always take the output of Subscriber.mailman_key() and apply lower to it, everywhere. That should only apply to the files that use dlists only (I hope only DlistUtils.py, but you'll need to grep for mailman_key to make sure you find them all.)

Kanika vats, do you have time to fix this (it will be faster for you to do it than me, as you have the right environment set up)?

[kanika vats (kanika-krikan)]

Yes sure I will fix the bug and commit to Systers stable branch soon.

[kanika vats (kanika-krikan)]

I have committed the changes to the stable branch at lp:~systers-soc/systers/stable.

The functions that received lowercase email address (even though the email-id was stored in uppercase in database) are:

1)getSubscriber_id_raw(self, addr) : addr is in lowercase
2)getSubscriber_id_raw_or_die(self, addr) : addr is in lowercase
3)setDisable(self, member, flag) : member is in lowercase
4)setDigest(self, member, flag) : member is in lowercase
5)changePreference(self, member, preference) : member is in lowercase
6)changeFormat(self, member, format) : member is in lowercase
7)changeAddress(self, oldaddr, newaddr) : oldaddr is in lowercase but newaddr come out to be uppercase
8)unsubscribeFromList(self, key) : here email id is recieved in lowercase
9)subscribeToList(self, key) : here email id is recieved in uppercase
10)mergeSubscribers(self, oldaddr, newaddr) : oldaddr is in lowercase and newaddr is in uppercase

Again want to mention that when I write " 'x' is in lowercase " I mean that even though the mail id was stored as uppercase in database but the functions get lowercase email-id as their input parameter.

I have made the necessary changes to handle the above issues but was not able to test whether members with uppercase email-id are able to post easily,create new thread or continue existing ones or not as locally I need to create users on my ubuntu and I found that ubuntu does not allows us to create users with uppercase letters in it.So I leave this part of testing on the testing rounds we will be conducting.