thunderbird email search should begin with the most recent email then progress to the oldest (what the user is searching for is more likely to be in recent rather than older emails)

[DrKay]

This becomes more problematic the more email a user has to search. With a large account, this can result in very long search times, which users are more likely to understand if the email is old. If it is a recent email, users expect to see the search result more quickly.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: thunderbird 7.0.1+build1+nobinonly-0ubuntu1
ProcVersionSignature: Ubuntu 3.0.0-12.20-generic 3.0.4
Uname: Linux 3.0.0-12-generic x86_64
AddonCompatCheckDisabled: False
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: PCH [HDA Intel PCH], device 0: CONEXANT Analog [CONEXANT Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: james 1977 F.... pulseaudio
BuildID: 20110929183320
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'PCH'/'HDA Intel PCH at 0xd2520000 irq 44'
   Mixer name : 'Intel CougarPoint HDMI'
   Components : 'HDA:14f1506e,17aa21da,00100000 HDA:80862805,80860101,00100000'
   Controls : 20
   Simple ctrls : 8
Card29.Amixer.info:
 Card hw:29 'ThinkPadEC'/'ThinkPad Console Audio Control at EC reg 0x30, fw unknown'
   Mixer name : 'ThinkPad EC (unknown)'
   Components : ''
   Controls : 1
   Simple ctrls : 1
Card29.Amixer.values:
 Simple mixer control 'Console',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Channel: release
Date: Wed Nov 2 20:02:56 2011
EcryptfsInUse: Yes
ForcedLayersAccel: False
IfupdownConfig:
 auto lo
 iface lo inet loopback
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
IpRoute:
 default via 192.168.1.1 dev wlan0 proto static
 169.254.0.0/16 dev wlan0 scope link metric 1000
 192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.102 metric 2
Prefs:
 places.database.lastMaintenance - 1320169182
 extensions.lastAppVersion - 7.0.1
 network.cookie.prefsMigrated - true
 gfx.blacklist.suggested-driver-version - Mesa 7.10.3
 places.history.expiration.transient_current_max_pages - 121141
Profiles: Profile0 (Default) - LastVersion=7.0.1/20110929183320 (Running)
RunningIncompatibleAddons: False
SourcePackage: thunderbird
UpgradeStatus: Upgraded to oneiric on 2011-10-17 (17 days ago)
dmi.bios.date: 04/01/2011
dmi.bios.vendor: LENOVO
dmi.bios.version: 8DET42WW (1.12 )
dmi.board.asset.tag: Not Available
dmi.board.name: 4286CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr8DET42WW(1.12):bd04/01/2011:svnLENOVO:pn4286CTO:pvrThinkPadX220:rvnLENOVO:rn4286CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 4286CTO
dmi.product.version: ThinkPad X220
dmi.sys.vendor: LENOVO

[Pawel-pohl (pawel-pohl)]

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5 FirePHP/0.2.1
Build Identifier: 2.0.0.17

I have migrated to Thunderbird from Outlook because Outlook's search feature was too slow. I have imported a database with 2 years worth of emails. Thunderbird's search is faster, but it starts with the *oldest* messages; that means that for full body searches (ex.: body contains: "new ftp password") I have to wait 5+ minutes to get to newer (more relevant) messages.

I know that I can limit the messages to, say, last 100 days or 400 days; however, I usually don't know how many days ago the relevant message was sent. Besides, that is an ugly and non-elegant solution.

Reproducible: Always

Steps to Reproduce:
1. Get a big email database
2. Start either a simple or advanced search using "body contains".
Actual Results:
The search takes a long time (which is OK given the data size), result list is updated on-the-fly (which is very good), but oldest results appear first (which is undesired).

Expected Results:
The results appear starting from most recent messages.

I am doing a local search, not an IMAP search.

I am marking this bug/feature request as "major", because it makes one of the most used functions (search) unusable in my case.

[Ludovic-mozilla (ludovic-mozilla)]

*** This bug has been marked as a duplicate of bug 439697 ***

[Ludovic-mozilla (ludovic-mozilla)]

Steps to Reproduce:
1. Get a big email database
2. Start either a simple or advanced search using "body contains".
Actual Results:
The search takes a long time (which is OK given the data size), result list is
updated on-the-fly (which is very good), but oldest results appear first (which
is undesired).

Expected Results:
The results appear starting from most recent messages.

[Pawel-pohl (pawel-pohl)]

Ludo(In reply to comment #1)
>
> *** This bug has been marked as a duplicate of bug 439697 ***

This is a completely different bug. Note that I am not complaining about the ordering of the display list. I am complaining about the order in which the messages are *searched*.

I am very disappointed to see this bug unresolved for so long. I have just come back to this issue after trying to find another message from three weeks ago. I entered the search term, and the search started about 5 minutes ago. I managed to navigate here and find this thread and write this comment, and the search result page is only half-way through 2008. Please, please change the order in which the messages are *searched*. Searching is not an instantaneous procedure, it takes a lot of time, which is why I presume the search results list is updated live.

I marked this as "polish" because, while I am ignorant to the underlying code, I suspect that it will only require a small change in the searching code. Correct me if I am wrong.

[Ludovic-mozilla (ludovic-mozilla)]

Search is being completely rewritten in TB 3.0 - the new functionality should be released pretty soon - we will make announces mdat and on blogs - give it a go and tell us if it solves your issue.

[Pawel-pohl (pawel-pohl)]

Thanks, I will make sure to update as soon as Tbird 3 is out!

[Euryalus-0 (euryalus-0)]

*** Bug 645984 has been marked as a duplicate of this bug. ***

[Euryalus-0 (euryalus-0)]

*** Bug 647828 has been marked as a duplicate of this bug. ***

[Emilis Dambauskas (emilis-d)]

I am using Thunderbird 3.1.8 on Ubuntu and this is not solved yet.

I see this bug registered for version 2.0. Should I register a new bug?

[Phil. V (philip.v)]

This sounds like a good idea to optimize search times, but this is more of a feature request and I suspect we should avoid an Ubuntu delta for it. Would you mind opening a bug/feature request against Thunderbird on Bugzilla for this? That way the Thunderbird developers can determine if this can be incorporated into their search algorithm, and if it would benefit the majority of users, as it's possible the current behavior is based on usage data showing that, on average, users search for old mail more often than recent mail.

[DrKay (dr-jameskay)]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20111008085652

Steps to reproduce:

I searched for a word in my email Inbox

Actual results:

I noticed that Thunderbird searches my Inbox starting from the oldest email (from several years ago) to the newest email, which makes searching take a long time.

Expected results:

It should begin searching with the most recent email, because most of the time when I search my email Inbox, I am searching for something recent (not years ago). If I am searching for something older, I would understand a longer wait time.

[DrKay (dr-jameskay)]

No problem. I've filed a bug/feature request against Thunderbird in Bugzilla. It's bug#699392.

https://bugzilla.mozilla.org/show_bug.cgi?id=699392

[Phil. V (philip.v)]

Thanks for doing that. I went ahead and linked the bugs on Launchpad.

[Mkmelin+mozilla (mkmelin+mozilla)]

*** This bug has been marked as a duplicate of bug 474730 ***

[Mkmelin+mozilla (mkmelin+mozilla)]

*** Bug 699392 has been marked as a duplicate of this bug. ***

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Davidhodges-nz (davidhodges-nz)]

Still not fixed more than two years later in Thunderbird 8 ! Please please fix this!

[G-sba (g-sba)]

I am using TB 12 and its still not solved! Please fix this!

[Educmale (educmale)]

In ver 15, this still executes the search from oldest to newest. Evil, with large data sets when searching through the body [for text].

May I suggest either a default direction reversed, into Ver 16 or 17, or allow for directional suggestions in the UI

Couldn't be a huge change, here, could it ?

Can anyone pinpoint the code where the search is called . . . I'll take a look, if someone gives me a starting point.

[Educmale (educmale)]

Still "upside down" search order in Ver 16.0.1

[Mkmelin+mozilla (mkmelin+mozilla)]

Created attachment 671123
proposed fix

Simple one-liner

[Mark David (mhd-yv)]

I would ideally like to see it search in the order of display. E.g., could be by 'from' from z to a, assuming you'd clicked the 'from' header once or twice effect that order. Then you would not need a separate option. In the short run, I'd settle for just searching in date order from newest to oldest.

[Educmale (educmale)]

Mark raises a good point -- Magnus: can you make the call to [Reverse]EnumerateMessages follow the users ordering....?

might be a bit of complexity if the user's sort order isn't by date?

[Mkmelin+mozilla (mkmelin+mozilla)]

Yes that would be more complex, and the use cases for it are not that many - then you'd need to know e.g. who it's from and that that who happens to be in a suitable place in the alphabet. And for date we shouldn't do it anyway, since you might very well want to sort the messages in one order, but in most cases what's newest is more important.

[Standard8 (mbanner)]

Comment on attachment 671123
proposed fix

Ok, I don't see any issues with this. r=me.

[Mkmelin+mozilla (mkmelin+mozilla)]

http://hg.mozilla.org/comm-central/rev/ed60a8f558aa -> FIXED

[Mkmelin+mozilla (mkmelin+mozilla)]

Backed out since this seems to have caused some oranges -- https://tbpl.mozilla.org/php/getParsedLog.php?id=16737186&tree=Thunderbird-Trunk

http://hg.mozilla.org/comm-central/rev/f0a1dc6af47c

[Mkmelin+mozilla (mkmelin+mozilla)]

Also to check - https://tbpl.mozilla.org/php/getParsedLog.php?id=16736733&tree=Thunderbird-Trunk&full=1

[Mkmelin+mozilla (mkmelin+mozilla)]

(In reply to Magnus Melin from comment #21)
> Also to check -
> https://tbpl.mozilla.org/php/getParsedLog.php?id=16736733&tree=Thunderbird-
> Trunk&full=1

This one wasn't related, went away before the backout.

[Mkmelin+mozilla (mkmelin+mozilla)]

The test failure is always reproducible, but i don't think i see a real world bug (though the refresh isn't doable the same way from the ui afaik). The failure is here: http://mxr.mozilla.org/comm-central/source/mail/base/test/unit/test_viewWrapper_virtualFolder.js#350

Andrew, any idea about this?

[Bugmail-asutherland (bugmail-asutherland)]

(In reply to Magnus Melin from comment #23)
> Andrew, any idea about this?

I accuse the reverse enumerator of stopping prematurely, skipping the first header in the box, but the quick perusal I did of the code in question was not enough to make it obvious where the glitch is. I would suggest looking into the enumerator more.

[Mkmelin+mozilla (mkmelin+mozilla)]

Created attachment 724560
proposed fix, v2

Try looks happy with this (except for the usual oranges) - https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cf1e60fd4c8a

[Ludovic-mozilla (ludovic-mozilla)]

Comment on attachment 724560
proposed fix, v2

I'm pretty It's :asuth that you wanted as a reviewer - switching.

[Mkmelin+mozilla (mkmelin+mozilla)]

Thx, must have misclicked.

[Stéphane Guillou (stephane-guillou)]

I have this problem on Ubuntu 13.04 64 bit with Thunderbird 17.0.6
For some reason, I can not remember encountering it when I was using Ubuntu 12.10 (not sure if the TB version was different...)

It is fairly frustrating as 99% of the time it the most recent result that I am looking for, so I have to wait and see emails dating back to 2009 being listed as results.

[Educmale (educmale)]

Any chance that :asuth might look see in on this ?

[Mkmelin+mozilla (mkmelin+mozilla)]

Andrew? ping

[Bugmail-asutherland (bugmail-asutherland)]

Comment on attachment 724560
proposed fix, v2

r=asuth, tremendously sorry about the delay.

[Mkmelin+mozilla (mkmelin+mozilla)]

https://hg.mozilla.org/comm-central/rev/d195627d5482 -> FIXED

[Educmale (educmale)]

Out of curiosity, Magnus, when is v24.0 actually scheduled...? My up-to-date version is only at 17.0.6, now....

(Or: Is there a reason why such fixes aren't merely inserted into the next release, say, 17.0.7 ?)

I ask, because this patch could be immediately useful.

[Educmale (educmale)]

(In reply to john ruskin from comment #32)
.... say, 17.0.7 ?)

Or to any next available version, before 24, before 3 months from now? My vote would be to proffer it to the next release.

I plead ignorance about how the versioning system works, schedules, and etc., and the system for choosing the version into which a patch is placed.

[Mkmelin+mozilla (mkmelin+mozilla)]

It's available in nightly builds already if you're in a hurry and willing to accept some rough edges. The next non-developer version is tb24, which will be released in september - https://wiki.mozilla.org/Releases#Next_Thunderbird_Major_Release

These kind of fixes aren't backported to the stable branch (tb 17), as patches can always cause unwanted side effects nobody thought they would.

[Jim (squibblyflabbetydoo)]

*** Bug 886122 has been marked as a duplicate of this bug. ***

[Paul White (paulw2u)]

Upstream bug showing "RESOLVED FIXED" on 2013-06-07
Target release - Thunderbird 24.0
Checked ok in Ubuntu 18.04 and Thunderbird 60.2.1
Marking as "Fix Released" to close