Nickname not over-riding names in email address

[Robin]

From what I have read in the support forum the nickname should take priority in the search for an email address in the to: field. It does not appear to work any more.

Example:

One friend's address is Mark<email address hidden>. The other is James<email address hidden>. Both are in my address book with Mark entered as the nickname of <email address hidden> and James as nickname for <email address hidden>.

If I type james in the To: field I get james.mark as a first entry in the list of available addresses.

If I type mark I get the same address again, when I would prefer the first optional address to be <email address hidden>

Reason it is needed:
I am trying to avoid sending emails meant for <email address hidden> to James.mark because I do not notice in time that I have defaulted to the first entry in the list of available addresses.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: thunderbird 10.0.2+build1-0ubuntu0.11.10.1
ProcVersionSignature: Ubuntu 3.0.0-16.29-generic-pae 3.0.20
Uname: Linux 3.0.0-16-generic-pae i686
ApportVersion: 1.23-0ubuntu4
Architecture: i386
BuildID: 20120216123548
Date: Fri Mar 16 00:31:06 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
SourcePackage: thunderbird
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian-atkins (brian-atkins)]

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5
Build Identifier: Thunderbird 1.5

I have an address book entry with the nickname "wil". When I type "wil" in the To line in the composition window, the result is an entry who's last name is "Wilbur" and which has no nickname at all.

The algorithm for finding addresses in the phone book should have the highest precedence on a full string match of nickname (after all, why did the user take the time to enter a nick name after all"). Imho, the precedence should be:

1) Full string match on nickname
2) Full string match on last name
3) Full string match on first name
4) Initial string match on nickname
5) Initial string match on last name
6) Initial string match on first name

You may not think this is important, but anything that causes mail to get sent to the wrong user inadvertantly is just about he worst possible problem a mailer can have.

Reproducible: Always

Steps to Reproduce:
1. Create an entry with "wil" as the nickname
2. Create an entry with Wilber as the last name
3. Enter "wil" in the To: field of the composition window

Actual Results:
The entry with the last name Wilbur is autocompleted

Expected Results:
The entry with the nickname "wil" should have been autocompleted

I entered this defect before, and have seen others report it. However, I searched Bugzilla and, searching for "nickname", did not find it. I apologize if this is duplicate, but after about 30min searching, I couldn't find the old bug.

[Brian-atkins (brian-atkins)]

I should have added a possibly important point. I have multiple address books, which appear, and I assume are searched in alphabetical order. The entry for "Wilber" appears in an address book at the top, which begins with "A". The entry with "wil" as the nickname appears in an address book further down, which begins with "I".

Perhaps the solution is to allow the user to set the order by which the address books are searched?

[Mcow (mcow)]

See bug 323364, which references several other bugs about list ordering.

I'm not convinced this is a major bug; it sounds like an enhancement request
to me.

[Eg-pat-tx (eg-pat-tx)]

(In reply to comment #0)
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8)
> Gecko/20051111 Firefox/1.5
> Build Identifier: Thunderbird 1.5

This anomoly appeared after I updated to v1.5 Before typing "Jim" followed by the ENTER key produced the ONE email with nickname "Jim". Now my mail is getting addressed to an email address beginning with "Jim..." Why on earth would anyone updating the app change this behavior. I can't use nicknames at all now because they DON"T WORK!

[Stachnis (stachnis)]

> This anomoly appeared after I updated to v1.5 Before typing "Jim" followed by
> the ENTER key produced the ONE email with nickname "Jim". Now my mail is
> getting addressed to an email address beginning with "Jim..." Why on earth
> would anyone updating the app change this behavior. I can't use nicknames at
> all now because they DON"T WORK!

I totally agree. I send several email sto the wrong person in the last days.
I changed back to Mozilla Suite, which does not has not this in my opinion
really serious bug.

[Alexandre-ferrieux (alexandre-ferrieux)]

Total agreement, this is a serious *regression* from ol'good Mozilla 1.7.
And also from any decent mail client by the way: only an idiot would prefer to have all substring matches from LDAP or collected address book take precedence over the hand-crafted, ultra-short nicknames.

So currently for me Thunderbird is unusable, because of theis very nickname issue.

Since this bug was reported 8 months ago, and judged "just a feature request" by the experts, I'm going to drop Thunderbird, and avise many others to think twice before leaving mozilla.

[James Teh (jteh)]

Regardless of whether this is an enhancement request or a bug, I think it is an important issue. If I assign a nickname to an entry in the address book, this means i want to refer to that user by that nickname. Other matches should still show in the autocomplete list, but I do think that the entry with matching nickname should take precedence.

[Standard8 (mbanner)]

The current TB trunk (3.x) builds, and I believe TB 2.0 have been changing to sort by how much you use particular addresses (essentially "popularity"). IMHO this is far better than sorting by nickname, however Bryan may have some thoughts on this.

[Mcow (mcow)]

Somewhere I described this example of why nickname should trump popularity:

My mother is named Mary. I have a friend named Mary. Mom's AB entry has a nickname "Mom" -- and I write her far more often than I do my friend. My friend's AB entry has a nickname "Mary". But if I open a compose window and type Mary, it's my Mom who gets the first match because she's "more popular."

That's just wrong. Nicknames should take highest priority.

xref bug 331817

[Eg-pat-tx (eg-pat-tx)]

Currently "nickname" in the address book does not work. I am really annoyed that this bug was never considered serious enough for someone to fix. Until it is, someone should ELIMINATE the nickname option. It is dangerous for people typing a "nickname" in the address "To" assuming it will actually work.

[Eagle-lu (eagle-lu)]

Created attachment 342561
patch

set PopularIndex based on the different matching case.

[Standard8 (mbanner)]

Comment on attachment 342561
patch

Sorry, this just isn't the right fix.

One of the problems is it ignores the popularity index (i.e. overwrites it).

I think what we really need here for this bug, is to consider our current algorithms and how we can improve them. I'm hoping Bryan (our User Experience guy) can set up a discussion on that.

[Bryan Clark (clarkbw)]

This sort order is not going to work effectively across all the different ways that people are searching for others. One type of use requires the nickname is more important and the other type requires first or family names.

The real goal here is that when a user types 'wil' into the auto-complete for the first time we show all the possible matches on different (and relevant) fields for that term. When the user selects the desired person from the auto-complete list Thunderbird associates the term 'wil' with that person.

This idea is called 'frecency', by the Firefox team, which was built into the FF3 awesomebar auto-completion. It gives the user a sense of the application learning their habits and allows them to find the same person by the same string over and over again.

We need to have a similar algorithm to this built against our contact database. I believe Gloda already has an idea of 'frecency' built into it, however we need to keep tweaking the data points it uses.

https://developer.mozilla.org/En/The_Places_frecency_algorithm

We should probably open a new bug for a frecency index and I recommend against this path.

[James Teh (jteh)]

(In reply to comment #12)
> This sort order is not going to work effectively across all the different ways
> that people are searching for others. One type of use requires the nickname is
> more important and the other type requires first or family names.
I don't understand when first or family names should ever be higher precedence than nickname. As I understand it, nickname should override Thunderbird's "frecency"; it essentially indicates that a user wants to always associate a specific word with a contact and that this word should always be used to gain quick access to this contact.

[Eagle-lu (eagle-lu)]

(In reply to comment #11)
> (From update of attachment 342561 [details])
> Sorry, this just isn't the right fix.
>
> One of the problems is it ignores the popularity index (i.e. overwrites it).
>
searching "PopularityIndex" in mxr.mozilla.org, I can't find anywhere else that use it expect here. What's the meaning of " PopularityIndex"? Isn't it how "popular" the AB item is?

[Standard8 (mbanner)]

(In reply to comment #14)
> (In reply to comment #11)
> > (From update of attachment 342561 [details] [details])
> > Sorry, this just isn't the right fix.
> >
> > One of the problems is it ignores the popularity index (i.e. overwrites it).
> >
> searching "PopularityIndex" in mxr.mozilla.org, I can't find anywhere else that
> use it expect here. What's the meaning of " PopularityIndex"? Isn't it how
> "popular" the AB item is?

Its values are set up in nsMsgCompose.cpp it is designed to be incremented each time the user sends a message to a contact. There are problems with it (for instance it just increments, there's no frequency option in there) which are covered in different bugs. The only actual use currently is in the autocomplete code.

[Finishlynx (finishlynx)]

Just something to try to drag this back to fundamentals here.

As a Luddite who just the other day "upgraded" from Netscape Mail 7.2 to Thunderbird, this bug is the first thing I noticed (yes, it is a bug, in the category of "something that used to work and is now broken"). To be fair, I was happy to discover just how similar Thunderbird was to my old favorite clunker 7.2. But this bug drives me crazy. It is *way* too easy to improperly address email by mistake. No, I don't want the software "thinking" for me about who "pete" is based on popularity or the alphabet or anything else. I know who "pete" is, I have told the software who "pete" is, and that is what matters.

I would say that the only reason that there hasn't been more noise about this is because it is not obvious to the average user (like me) just how one goes about complaining about something like this. It took me a long time to find this forum. But here are some others I found along the way that show we-who-would-like-this-fixed are not outliers.

http://fixunix.com/mozilla/435688-nickname-priority.html
http://forums.mozillazine.org/viewtopic.php?f=39&t=371872

Thanks for listening.

[Eg-pat-tx (eg-pat-tx)]

Keep on shouting until someone fixes this bug. It's bad design to offer nicknames, then place them bottom priority in the UI. I sent mail to the wrong person without noticing the bug when I switched to Tbird. Nearly went back to my previous mailreader and can't believe it's still unresolved!

You couldn't have said it better: "... this bug drives me crazy. It is *way* too easy to improperly address email by mistake. No, I don't want the software "thinking" for me about who "pete" is based on popularity or the alphabet or anything else. I know who "pete" is, I have told the software who "pete" is, and that is what matters."

[9e9o1ko8b2f5xpiibg-chris-0zxvj9hhx1hzo5xiyh (9e9o1ko8b2f5xpiibg-chris-0zxvj9hhx1hzo5xiyh)]

I just installed TB 3 and because the address popularity counts seem to have been cleared out in the process, the lack of nickname precedence is even more noticeable - I may just have to turn off autocomplete altogether so I stop embarrassing myself with messages sent to the wrong recipient. :) It would be great to see a fix for this soon. Bug 530865 seems to be related. Thanks!

[Bryan Clark (clarkbw)]

(In reply to comment #17)
> Keep on shouting until someone fixes this bug.

Please do not, this is not how effective communities can work. I can understand wanting changes to happen (I have bugs in Firefox that aren't fixed) but working together as well behaved adults is what makes progress.

If we had a patch that merely weighted the nickname matches above the popularity index then I think that would be an acceptable update to our current system. I believe a frecency index would be a much better value to use however that's for another bug to tackle.

[Bryan Clark (clarkbw)]

@Chris, thanks I hadn't seen the seamonkey version, bug 530865

[Themoz (themoz)]

An earlier commenter said they were a Luddite who only recently upgraded to Thunderbird 3. Count me among the same crowd. I had been using a shareware mail client from roughly 2002.

That client allowed me to address an e-mail by typing a comma-delimited list of nicknames into any of the relevant fields (TO, CC, or BCC). Thunderbird also supports parsing a comma-delimited list of email addresses into multiple entries into its addressing user interface.

Expanding on the example above, assume you have nicknames "wil", "jan", and "bob". You should be able to type "wil,jan,bob" to quickly address a mail to all three. When focus leaves the field, the parser should expand each based on a nickname match, resulting in three entries in the addressing user interface.

The current behavior will break apart the comma-delimited list into three entries, but the nicknames will be retained (that is, the user interface will literally say "To: wil", "To: jan", and "To: bob").

It seems the current addressing design in Thunderbird is to attempt to resolve the address you're typing before you leave the current text field. This explains the "(user input) >> (resolution proposal)" user interface gimmick in the text input field.

I think that hitting tab when a proposed resolution is displayed--even if only momentarily or perhaps as an event that fires prior to losing focus--acts as a user selection of the proposed resolution. The trouble is that parsing of a comma-delimited list is apparently executed after that resolution process. The resolution process finds no match for "wil,jan,bob" and therefore assumes that represents the totality of the desired input and passes the value over to the comma-delimited parser.

I consider nicknames to be a special case of address resolution. As the proponents above have said, at the least nicknames should be treated as the #1 priority in finding a matching address. But I would recommend going further: namely, nickname resolution should occur -after- the comma-delimited parsing of the user's input occurs.

[Eg-pat-tx (eg-pat-tx)]

To repeat Brian's perfect comment after months patiently waiting for a solution:
"You may not think this is important, but anything that causes mail to get sent
to the wrong user inadvertently is just about he worst possible problem a
mailer can have."

[Y9a7s7tjd2jxyck-steve (y9a7s7tjd2jxyck-steve)]

How can this be classified as Enhancement.
https://bugzilla.mozilla.org/page.cgi?id=fields.html#importance
It is certainly a loss of function, so it's somewhere between Normal and Major. (and I say Major).
Other than soliciting votes (which I have done in Moz community support) how can one get the impotance correctly defined?

[Y9a7s7tjd2jxyck-steve (y9a7s7tjd2jxyck-steve)]

(In reply to comment #12)
> This sort order is not going to work effectively across all the different ways
> that people are searching for others. One type of use requires the nickname is
> more important and the other type requires first or family names.
>
What other use of nicknames is there other than addressing messages? I don't follow the notion that one might want another field to have higher priority in this context.

[9e9o1ko8b2f5xpiibg-chris-0zxvj9hhx1hzo5xiyh (9e9o1ko8b2f5xpiibg-chris-0zxvj9hhx1hzo5xiyh)]

I've inquired about getting this bug reclassified as a loss of functionality:

http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/11146031bdc47ecd#

Chris

[Nikolay Shopik (shopik)]

Who experience this regression in TB3 should have look into bug 497722.

[Bryan Clark (clarkbw)]

*** Bug 450279 has been marked as a duplicate of this bug. ***

[Kam3bombh9k-140-wph05ilav2k (kam3bombh9k-140-wph05ilav2k)]

bug 497722 is not the same as this bug. Further, I installed 3.01, and bug 497722 is not fixed. Entering a letter into the "to" field still produces random results.

[Standard8 (mbanner)]

(In reply to comment #28)
> bug 497722 is not the same as this bug.

Correct.

> Further, I installed 3.01, and bug
> 497722 is not fixed.

Yes it is not fixed in 3.0.1 as it was fixed after 3.0.1. It will be fixed in 3.0.2 (not released yet).

[Kam3bombh9k-140-wph05ilav2k (kam3bombh9k-140-wph05ilav2k)]

My brief testing indicates that trunk build 3.2a1pre has fixed this bug. The "to" field now fills in like it did under Tbird 2.0.

[Vseerror (vseerror)]

*** Bug 530865 has been marked as a duplicate of this bug. ***

[Kzqdnsw4i20a5p-68p7m-x8oxkp4um1x7tv (kzqdnsw4i20a5p-68p7m-x8oxkp4um1x7tv)]

This bug is still present as of 3.1.7 - ie. Use of a unique nicknames doe not result in the associated email address appearing as the first item in the list of suggested email addresses after the autocomplete function runs. I suggest tha in the tab:

  Tools -> Options -> Composition -> Addressing:

an option of the form:

  [ ] Search Nickname field first?

.. be added, and of course the code updated to work that way.

[Peter-cherna (peter-cherna)]

I agree this is a serious misbehavior and as far as I can tell (from the end-user point of view, not from source), this is a regression. In Thunderbird 2, if I typed "Bob", and the auto-completion served up a name I did not prefer, I could go into the address book, add "Bob" as the nickname for my favorite Bob, and thereafter typing "Bob" would *always* complete to the desired one. When I moved to Thunderbird 3.1, this was lost. (Still true as of 3.1.7)

I don't know what other purpose nicknames even serve, if not to be used for completion.

As far as I can tell from reading the various here-linked bugs on auto-completion order, none of them address the apparent regression in nickname handling.

(I think a future change to use frecency might work out, and one might argue that a nickname should give weight in frecency much like bookmarking a URI might, but given this is seems to be a regression, and it's serious enough (misdirected email), I would hate to see the fix for this entangled into a larger issue of frecency.)

[Eg-pat-tx (eg-pat-tx)]

6 years is far too long to either:
(a) Remove the nickname from the email options
(b) Make it work (when I type Ann followed by a carriage return, it automatically fills in the address for my nickname Ann - and no other!

[Erich-nospam+bugzilla (erich-nospam+bugzilla)]

Just sent an email to the wrong person due to this bug. Here's the situation:

I have the following two people in my Personal Address Book:
First: Zhichao
Last: Zhang
Display: Jack Zhang
Nickname: Jack
Email: <email address hidden>

First: Jack
Last: Lastname2
Display: Lastname2, Jack
Nickname: (blank)
Email: <email address hidden>

My email address is <email address hidden> (same company as first Jack)

I occasionally send email to Jack from Company 2, but (obviously since I've added a nickname for Jack number 1) when I type "Jack" in the To: field, I want email to go to the Jack in my own company. As you can see, he has taken an American name (Jack) to go by instead of his real name (Zhichao).

Well, when I type "Jack" in the To: field, Thunderbird only gives me one option, Jack from Company 2! If I want to send email to Jack #1 I have to type "zh" and select his name from the multiple choices, or type "zzc" and his email is the only choice. It seems Thunderbird completely ignores the nickname, as it doesn't even offer Jack Zhang.

Ok, further testing reveals if I type slowly ("Jac", then wait) both Jacks are provided as options. If I then further type the "k", both Jacks remain as options for me to choose from.

I would like AT A MINIMUM for both Jacks to be options if I type "Jack" quickly, and of course I agree with everyone above that Jack Zhang should be the first option when typing "Jack".

[Kreutz-9 (kreutz-9)]

Isn't there any way to get this bug fixed? It's so obviously idiotic, and leads to such embarrassing and/or inconvenient results.

Help!!!!

[Manuel López-Ibáñez (manuellopezibanez)]

(In reply to Tom Kreutz from comment #36)
> Isn't there any way to get this bug fixed? It's so obviously idiotic, and
> leads to such embarrassing and/or inconvenient results.
>
> Help!!!!

It would be nice if Mozilla Messaging provided a convenient way to crowd-fund specific features. Something simpler than kick-starter and perhaps connected to bugzilla.

[Bugzilla2007 (bugzilla2007)]

(In reply to Brian Hauer from comment #21)
> Expanding on the example above, assume you have nicknames "wil", "jan", and
> "bob". You should be able to type "wil,jan,bob" to quickly address a mail
> to all three. When focus leaves the field, the parser should expand each
> based on a nickname match, resulting in three entries in the addressing user
> interface.
> [snip]
> I consider nicknames to be a special case of address resolution. As the
> proponents above have said, at the least nicknames should be treated as the
> #1 priority in finding a matching address. But I would recommend going
> further: namely, nickname resolution should occur -after- the
> comma-delimited parsing of the user's input occurs.

+1 to all of comment 21 and the underlying understanding of nicks as 1 on 1 aliases. However, focus in this bug is on giving resolved (single) nicks precedence in autocomplete results. So resolving multiple comma-separated nicks is probably beyond the scope of this bug. More reminiscent of (but perhaps not the same as) bug 295428. I'd recommend filing a separate bug for resolving a comma-separated list of nicknames.

[Bugzilla2007 (bugzilla2007)]

OT: (In reply to :aceman from comment #68)
> > > Now I can't send to a list!
> > j.mccranie, problem appreciated, but it's not related to this bug so let's
> > be focused. Feel free to file a new bug if this isn't on record yet. But I
> > think I've seen a recent bug for that which I can't find right now.
>
> Should have been fixed in TB31.1.1 via bug 1060901.

Thanks for the reference on that mailing list bug, and from there, the other one I had in mind was Bug 1008718.

[Bugzilla2007 (bugzilla2007)]

Aceman, Suyash, sorry for nagging, but given the added importance of this bug for the current autocomplete UX, could you answer to my Comment 59?

[Acelists (acelists)]

The infrastructure created in bug 970456 should make it easy to prioritize anything we want. Unless deciding about the priority of each contact is slow and so unnacceptable for large result sets.

[Bugzilla2007 (bugzilla2007)]

(In reply to :aceman from comment #71)
> The infrastructure created in bug 970456 should make it easy to prioritize
> anything we want. Unless deciding about the priority of each contact is slow
> and so unnacceptable for large result sets.

I want to believe so too, but aren't we sorting plain *results* there *after* the fact (aka email addresses, potentially two from each card), whereas for nickname we need to retrieve that at a much earlier point where we still have access to the card (and going back from email to card would be error-prone in current flawed design of AB)? In fact, since a card can have 2 mail addresses, both of them might have to be toplisted in case of nickname match, primary address first.

[Hoserjoe (hoserjoe)]

<rant> Such an incredible waste of resources! For years, I have puzzled about the use of the "Nickname" field, and now I see that it actually fits into someone's unbelievably obscure search algorithm that nobody in the known world has ever relied upon. Why anyone would go to all the work described here to organize each card to be searched by these esoteric methods is totally beyond reason.

All that is needed is a simple alpha index and search. If the developers are infatuated with exotic search methods to pad their resumes, fine, but leave us real world users with a simple alpha search. Getting carried away with complicated search methods as described above is complete nonsense!</rant end>

[Tilmann-reh (tilmann-reh)]

José, just because you don't know and/or use nicknames doesn't mean they are obscure, useless or don't make any sense. Particularly for me, nicknames (in most cases just the initials) are the (by far!) preferred and fastest way to enter a recipients address. Just enter two or three letters and pick the adress from the upcoming (very short) list. This has always worked perfectly fine for many years (since early Netscape), until it was broken due to some "enhancements" to the search function. We just want that to be repaired.

[Bugzilla2007 (bugzilla2007)]

(In reply to José Josephus from comment #73)
> <rant> Such an incredible waste of resources! [...] Getting carried away
> with complicated search methods as described above is complete
> nonsense!</rant end>

I've addressed the obvious social and factual shortcomings of that comment via private mail, including a link to https://bugzilla.mozilla.org/page.cgi?id=etiquette.html.

(In reply to Tilmann Reh from comment #74)
> José, just because you don't know and/or use nicknames doesn't mean they are
> obscure, useless or don't make any sense. Particularly for me, nicknames (in
> most cases just the initials) are the (by far!) preferred and fastest way to
> enter a recipients address. Just enter two or three letters and pick the
> adress from the upcoming (very short) list. This has always worked perfectly
> fine for many years (since early Netscape), until it was broken due to some
> "enhancements" to the search function. We just want that to be repaired.

+1. Thanks Tilmann.
We've heard your plea and we're working to repair that in Bug 970456, while preserving the actual "enhancements" of the search functions which we introduced to fix other bugs and cater for legitimate user expections.

[Acelists (acelists)]

All this talk is about the recipient autocomplete in the address widget. All the other searches (contacts sidebar, search in addressbook window and Edit->Search addresses should still sort alphabetically (by the column that is selected)). So no need to worry.

[Hoserjoe (hoserjoe)]

(In reply to :aceman from comment #76)
> So no need to worry.

Are you joking? None of the fancy nick-name, frequency searches work, the alpha searches produce gibbled results! The sidebar list is out of order. The whole thing is a broken down, useless wreck! A simple alpha search on "Display Name" would at least give us something to work with. Start listening with full attention, please!

[Acelists (acelists)]

Then please post it in the relevant bug, not one about autocomplete. Thanks.

[Syshagarwal (syshagarwal)]

I don't think I have to say anything now about this.
Thanks.

[Gilem (gilem)]

Exact match was the change that happened to me at some point in the past year. I have a few short nicknames like 'jim'. In previous version, typing quickly jim[tab] instantly prefilled from the nickname. In the newer versions, if there are any addresses in the book starting with jim, the behavior is to put a red 'jim' in the to box, and not allow sending at all. I fail to see the point of the nickname if it is not used, and one should not have to wait .5s for the autocomplete to prefill to hit tab.

[St7ve (st7ve)]

It's become a problem again in TB31.4 (or perhaps a little bit earlier) (with the visibly different compose window)

[Mkmelin+mozilla (mkmelin+mozilla)]

This should be pretty straight forward to fix, now that bug 970456 is fixed. You'd just adjust the scoring there to be higher if it's a match in nickname.

[St7ve (st7ve)]

I have submitted a new bug 1127258 which describes a new problem when selecting an address in TB31 (31.4); that's two addressing problems introduced with this version of TB which result in emails going to the wong person unless one takes special care.

[Bugzilla2007 (bugzilla2007)]

(In reply to Magnus Melin from comment #82)
> This should be pretty straight forward to fix, now that bug 970456 is fixed.
> You'd just adjust the scoring there to be higher if it's a match in nickname.

Suyash, could you try this? It's pretty important to offer a 100% predictable way of using autocomplete shortcuts... Magnus has volunteered to mentor this ;)

[Syshagarwal (syshagarwal)]

Created attachment 8556897
Patch v1

Oh, I thought it was supposed to be for someone looking for "getting started".
If its urgent, here's an attempt.

Thanks.

[Mkmelin+mozilla (mkmelin+mozilla)]

Comment on attachment 8556897
Patch v1

Review of attachment 8556897:
-----------------------------------------------------------------

::: mailnews/addrbook/src/nsAbAutoCompleteSearch.js
@@ +128,5 @@
> * Gets the score of the (full) address, given the search input. We want
> * results that match the beginning of a "word" in the result to score better
> * than a result that matches only in the middle of the word.
> *
> + * @param aCard - The card whose score is being decided

lowercase t?

@@ +139,5 @@
> +
> + // We will firstly check if the search term provided by the user
> + // is the nick name for the card or at least in the beginning of it.
> + let nick = aCard.getProperty("NickName", "");
> + nick = nick.toLocaleLowerCase();

you can just put .toLocaleLowerCase() after getPropery above

@@ +140,5 @@
> + // We will firstly check if the search term provided by the user
> + // is the nick name for the card or at least in the beginning of it.
> + let nick = aCard.getProperty("NickName", "");
> + nick = nick.toLocaleLowerCase();
> + aSearchString = aSearchString.toLocaleLowerCase();

remove the duplication of this below.

@@ +143,5 @@
> + nick = nick.toLocaleLowerCase();
> + aSearchString = aSearchString.toLocaleLowerCase();
> + let nickIndex = nick.indexOf(aSearchString);
> + if (nick == aSearchString || nickIndex == 0)
> + return BEST;

You don't need the nick == aSearchString
But, this doesn't quite do what this bug requests. What is requested is that nick should act as a "super-best" match. so the score should be 1 better than if it matched in display name - as is they would be equal (BEST + 1)

Please also adjust the test for this. There should be one test where the nick is the beginning of another persons name.

::: mailnews/addrbook/test/unit/test_nsAbAutoCompleteSearch6.js
@@ +139,5 @@
> card.setProperty("PopularityIndex", element.popularityIndex);
> card.firstName = element.firstName;
> card.lastName = element.lastName;
> + if (element.NickName)
> + card.setProperty("NickName", element.NickName);

You don't need the if I think. All the other props are also just set to undefined if not set. Also, please lowerCamelCase NickName

[Syshagarwal (syshagarwal)]

Created attachment 8557496
Patch v2

Made the suggested changes.

Thanks.

[Bugzilla2007 (bugzilla2007)]

Comment on attachment 8557496
Patch v2

Review of attachment 8557496:
-----------------------------------------------------------------

Thanks a lot Suyash for your quick response to my request to pick this up - I am really getting tired of referencing and explaining this bug (and workaround) in other bugs where users have been facing problems due to this major flaw which practically disables the entire nickname functionality for a range of scenarios.

I'll set feedback+ because we're on the right track here; but the devil - as always - in the detail (i.e. this patch as-is won't work yet as expected)...

::: mailnews/addrbook/src/nsAbAutoCompleteSearch.js
@@ +141,5 @@
> + // is the nick name for the card or at least in the beginning of it.
> + let nick = aCard.getProperty("NickName", "").toLocaleLowerCase();
> + aSearchString = aSearchString.toLocaleLowerCase();
> + if (nick.indexOf(aSearchString) == 0)
> + return BEST + 1;

(In reply to Magnus Melin from comment #86)
> You don't need the nick == aSearchString
> nick should act as a "super-best" match. so the score should be 1 better
> than if it matched in display name - as is they would be equal (=> we need BEST + 1)

True, but still not sufficient for ensuring absolute top ranking for *full* nickname matches over *partial* nickname matches, e.g.:
Card1: Nickname="foo" (nick==searchphrase; nickIndex==0)
Card2: Nickname="foobar" (nick!=searchphrase; nickIndex==0)
Searchphrase: "foo"

Autocomplete result (wrong):
Card2 (partial nick match)
Card1 (full nick match)

To make the nickname design work reliably, full nickname match must obviously supersede partial nickname match, so we would need to adjust the scoring accordingly:

if (nick == aSearchString)
  return BEST+2;
if (nickIndex == 0)
  return BEST+1;

However, I'm not convinced yet if making initial nickname matches rank higher than anything else will really work out well, e.g.:

Card1: Display Name="Tom" popularityIndex=10
Card2: Nickname="TomFoo" popularityIndex=5
Card3: Nickname="TomBar" popularityIndex=4
Searchphrase: "Tom"

Autocomplete result:
Card2 (partial nickname match, pi=5)
Card3 (partial nickname match, pi=4)
Card1 (full display name match, pi=10)

So I'm not sure if all results with partial matches on nickname should rank higher than full match on display name; should they? (I think not). Worse, PLEASE let's be aware that any forced top scorings that are not based on frecency will further undermine even the current rudimentary frequency implemtation of "popularityIndex":

Card1: Display Name="Tom" popularityIndex=1000
Card2: Nickname="TomFoo" popularityIndex=5
Card3: Nickname="TomBar" popularityIndex=4
Card4: Nickname="Tim" popularityIndex=6
Card5: Nickname="TD" popularityIndex=7
Searchphrase: "T"

Autocomplete result:
(all partial nickname matches top-listed, ordered by popularity:)
Card5
Card4
Card2
Card3
(then, potentially far down the list:)
Card1 (initial Display name match; popularityIndex=1000(!))

So in this case, even for partial nickname matches from single-character(!) search phrase, ALL partial nickname matches will be toplisted (although they are not popular at all, and the partial match on th...

[Bugzilla2007 (bugzilla2007)]

(In reply to Michael Gile from comment #80)
> and one should not have to wait .5s for the autocomplete to prefill to hit tab.

Michael, that's Bug 1012397.

[Bugzilla2007 (bugzilla2007)]

Nit: While we are here, can you please fix this comment:

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#152
> 152 // ":xx:", "jd", "who".
152 // ":xx", "jd", "who".

[Bugzilla2007 (bugzilla2007)]

(In reply to Thomas D. from comment #88)
> Comment on attachment 8557496
> Patch v2
>
> Review of attachment 8557496:
> -----------------------------------------------------------------
>
> True, but still not sufficient for ensuring absolute top ranking for *full*
> nickname matches over *partial* nickname matches, e.g.:
> Card1: Nickname="foo" (nick==searchphrase; nickIndex==0)
> Card2: Nickname="foobar" (nick!=searchphrase; nickIndex==0)
> Searchphrase: "foo"
>
> Autocomplete result (wrong):
> Card2 (partial nick match)
> Card1 (full nick match)

That result can occur e.g. when Card2 has a higher popularityIndex than Card1.
It shouldn't occur because full nickname matches (by inherent design) must supersede any other match, regardless of popularity.

[Mkmelin+mozilla (mkmelin+mozilla)]

Comment on attachment 8557496
Patch v2

Review of attachment 8557496:
-----------------------------------------------------------------

Hmm, Thomas has some point. So here's what I think we should do:

if (nick == aSearchString)
  return BEST+1;
if (nickIndex == 0)
  return BEST;

r=mkmelin, with that and test adjusted to pass (if needed)

[Syshagarwal (syshagarwal)]

Created attachment 8558325
Patch for check-in

Made the suggested changes.

Thanks.

[Bugzilla2007 (bugzilla2007)]

(In reply to Magnus Melin from comment #92)
> Comment on attachment 8557496
> Patch v2
>
> Review of attachment 8557496:
> -----------------------------------------------------------------
>
> Hmm, Thomas has some point. So here's what I think we should do:
>
> if (nick == aSearchString)
> return BEST+1;
> if (nickIndex == 0)
> return BEST;

Cool! Looks like it could be an elegant solution to problems mentioned in my comment 88. Thanks, great teamwork!

[Geoff (geoff-lankow)]

I've pushed this as https://hg.mozilla.org/comm-central/rev/5f277274b13f

[Bugzilla2007 (bugzilla2007)]

Magnus, can you please set the appropriate flags to get this into a TB release version as soon as possible, even for TB31 if possible?

I'm seeing more and more bug reports where users complain that nickname searches no longer work (they never have, but they appeared to... but now, depending on scenario / search words and data sets, users might see more irrelevant results, also because of irrelevant matches from collected addresses, resulting from our former mis-design (now fixed) to use collected addresses for remote-content permissions).

We should really land all autocomplete fixes fast, to avoid getting more unqualified complaints for things we have already fixed.

[Bugzilla2007 (bugzilla2007)]

*** Bug 1132681 has been marked as a duplicate of this bug. ***

[Mkmelin+mozilla (mkmelin+mozilla)]

I don't think this is ESR material as it's a new behavior not fixing a regression.

[Bugzilla2007 (bugzilla2007)]

(In reply to Magnus Melin from comment #98)
> I don't think this is ESR material as it's a new behavior not fixing a
> regression.

In view of 33 votes here and the quantity of bugs reported against autocomplete result sorting algorithm, I find that evaluation nitpicking. While in a strictly technical sense, this might not be a regression, it's actually worse:

1) Once upon a time we delivered a feature which apparently has never worked as intended and required by inherent design. That's a significant failure which shouldn't have to wait any longer when it's finally fixed 9 years after being reported.

2) We have only recently exposed this problem significantly more by expanding the scope of recipient search for certain scenarios. So, as seen from numerous reports, users are now more affected by this design problem, because of the changes made in recent version. Technicalities aside, that looks like a de-facto regression to me (as confirmed by users saying "it used to work until recently but now it fails").

I don't see any risks coming from this straightforward patch, except that we're undermmining popularityIndex a bit more again, but this works on exactly the same terms and conditions as Bug 970456 so there's nothing new here.

So I still think this 5-line-patch should land on ESR.

[Vseerror (vseerror)]

It is important to be nit-picky when one is in the role of deciding what to take on ESR, lest we get excessivly loosey goosey with our standards - which are looser than Firefox's, and thus have grey areas which necessitate judgement calls ultimately made by one person.

But I support the request for ESR uplift for one reason only, if comment 97 paragraph 2 is correct and the current net effect behavior to the user is it *appears* to be a regression - that's all that matters. So if patch has no known, and little to no likelihood of having, side effects, then I think we should take it. (Also, we have taken patches on ESR that do not fix recent regressions.)

[St7ve (st7ve)]

As one of the affected users, I want to plead with whoever has the clout to get this fixed as soon as, I find myself misaddressing messages practically every day since the latest release, which to me feels like a crime, to be honest it feels to me like something which needs fixing between releases

Going through the history of this, I spotted somone else's comment that the word nickname appeared on no less than 265 different threads on the getsatisfaction system, which sure sounds to me like a lot of people care (and will currently be in a state of disappointment)

Is there some sort of QA postmortem going on as to how this managed to get released in such a poor state? I don't mean to express (or provoke) anger with this point, I just think it's really useful to contemplate how the organisation has managed such a cock-up and how not to do it again.

Would it help if I volunteered to evaluate what is now proposed?

[Vseerror (vseerror)]

(In reply to steve hayes from comment #101)
> Going through the history of this, I spotted somone else's comment that the
> word nickname appeared on no less than 265 different threads on the
> getsatisfaction system, which sure sounds to me like a lot of people care
> (and will currently be in a state of disappointment)

I'm sorry to report that Getsatisfaction search (as well as the current SUMO search) is notoriously bad about giving false search hits "just because". So I'd doubt as many as 50 getsatifaction threads were solid hits about nickname autocomplete. On the current SUMO it is maybe about a dozen.

> Is there some sort of QA postmortem going on as to how this managed to get
> released in such a poor state? I don't mean to express (or provoke) anger
> with this point, I just think it's really useful to contemplate how the
> organisation has managed such a cock-up and how not to do it again.

The issues that got us here are pretty well understood by those involved. But probably the biggest hit to the end result (besides lack of personnel and competing priorities) was the lack of automated tests. Automated tests are in effect a spec - they define how the function should behave. Without automated tests, it's a crap shoot to know whether someone's new code functions worse, or better, than the old code. But lack of tests is not the fault of the current developers.

> Would it help if I volunteered to evaluate what is now proposed?

If you wish to help, there are many opportunties to give of your time (and time is what we most need). https://wiki.mozilla.org/Thunderbird#Contributing

[Mkmelin+mozilla (mkmelin+mozilla)]

As seen in the past every minor change might get someones use-case slightly wrong.
We have usually let patches bake through aurora+beta before esr, but with 38 coming up soonish I think what gets into 31esr should also be stricter.

Based on other reports, there may be something fishy happening regarding popularityindex, but I haven't had time to investigate that yet.

[Mkmelin+mozilla (mkmelin+mozilla)]

Comment on attachment 8558325
Patch for check-in

Review of attachment 8558325:
-----------------------------------------------------------------

We should land this for aurora+beta.

[Bugzilla2007 (bugzilla2007)]

Magnus, thanks for approving this for beta and aurora.

(In reply to Magnus Melin from comment #103)
> As seen in the past every minor change might get someones use-case slightly
> wrong.
> We have usually let patches bake through aurora+beta before esr, but with 38
> coming up soonish I think what gets into 31esr should also be stricter.
>
> Based on other reports, there may be something fishy happening regarding
> popularityindex, but I haven't had time to investigate that yet.

Well, apart from the fact that popularityIndex is broken by design (because it just counts up forever so doesn't consider recency), the "fishy" thing which happened to popularityIndex is that your patch with new scoring algorithm in Bug 970456 effectively disables it for an unpredictably large number of usecases where search string happens to match the visible results strings in a wordwise manner, more so for matching the beginning of the visible result string, which is typically the display name. So no matter now popular certain results are, they can never get to the top.

[Rkent-3 (rkent-3)]

Pushed to comm-aurora https://hg.mozilla.org/releases/comm-aurora/rev/13f0f3859462

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paul White (paulw2u)]

Upstream bug showing "RESOLVED FIXED" on 2015-02-03
Target milestone - Thunderbird 38.0
Confirmed working in Ubuntu 18.04 and Thunderbird 60.2.1
Closing by marking "Fix Released"