Beagle does not index Thunderbird mails

[Erdal Ronahi]

Although I have the beagle-extension in Thunderbird installed, Beagle never indexes my emails.

The output of beagle-info --index-info:

Name: Blam
Count: 0
Crawling: False

Name: Pidgin
Count: 0
Crawling: False

Name: KNotes
Count: 0
Crawling: False

Name: KOrganizer
Count: 0
Crawling: False

Name: Tomboy
Count: 27
Crawling: False

Name: EvolutionDataServer
Count: 767
Crawling: False

Name: Labyrinth
Count: 0
Crawling: False

Name: documentation
Count: 57534
Crawling: False

Name: KonqBookmark
Count: 0
Crawling: False

Name: EvolutionMail
Count: 66452
Crawling: False

Name: Thunderbird
Count: 32136
Crawling: False

Name: Liferea
Count: 0
Crawling: False

Name: Empathy
Count: 0
Crawling: False

Name: Opera
Count: 12
Crawling: False

Name: Kopete
Count: 0
Crawling: False

Name: IndexingService
Count: 0
Crawling: False

Name: NetworkServices
Count: -1
Crawling: False

Name: NautilusMetadata
Count: -1
Crawling: False

Name: KAddressBook
Count: 771
Crawling: False

Name: KonquerorHistory
Count: 2
Crawling: False

Name: manpages
Count: 4954
Crawling: False

Name: applications
Count: 677
Crawling: False

Name: KMail
Count: 0
Crawling: False

Name: Akregator
Count: 0
Crawling: False

Name: Konversation
Count: 0
Crawling: False

Name: Files
Count: 88444
Crawling: False

[Erdal Ronahi (erdalronahi)]

Complete removal and reinstallation helped, now after a while Beagle always crashes due to a SIGABRT bug in MONO:

#3 0x08bca270 in ?? ()
#4 0x00000005 in ?? ()
#5 0x0000438a in ?? ()
#6 0x08bca270 in ?? ()
#7 0x00000005 in ?? ()
#8 0xb7f9d5f8 in ?? () from /usr/lib/libglib-2.0.so.0
#9 0xb7f9d620 in ?? () from /usr/lib/libglib-2.0.so.0
#10 0xbfb6e37c in ?? ()
#11 0x00000001 in ?? ()
#12 0x00000001 in ?? ()
#13 0x0844a198 in ?? ()
#14 0x08bca270 in ?? ()
#15 0xb7e29b90 in ?? () from /lib/tls/i686/cmov/libc.so.6
#16 0xb7ed9df0 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#17 0xb7ed8520 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#18 0xb7f4c502 in g_thread_self () from /usr/lib/libglib-2.0.so.0
#19 0xb7f281e7 in g_main_loop_run () from /usr/lib/libglib-2.0.so.0
#20 0xb679ef91 in ?? ()
#21 0x084d7050 in ?? ()
#22 0xbfb6e404 in ?? ()
#23 0xb679ef56 in ?? ()
#24 0x0821cd78 in ?? ()
#25 0x00113fc0 in ?? ()
#26 0x00113fc0 in ?? ()
#27 0x000b3e40 in ?? ()
#28 0x00000000 in ?? ()
#0 0xb7fcd410 in __kernel_vsyscall ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted

[dBera (dbera-web)]

Disabling the EvolutionDataServer backend might help. There are similar crash bugs filed against evolution-sharp which beagle uses.

[Michele Mangili (mangilimic)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue for you. Can you try with the latest Ubuntu release? Thanks in advance.

[Erdal Ronahi (erdalronahi)]

I uninstalled Beagle and am not likely to use it again soon.

[Michele Mangili (mangilimic)]

Thank you Erdal for your quick response!

I'm going to mark your report as invalid, due to your last comment. Feel free to re-open it in case that you try Beagle again, and experience the same problem.

[Andreas Jonsson (sonofjon)]

I can confirm this bug. I'm running up-to-date Ubuntu Jaunty. I have three imap accounts. No email is indexed (see below).

I can't remove evolution-data-server, as suggested, since gnome-do-plugins and ekiga depends on it. I need these two packages. There is also evolution-data-server-common, which other important packages (e.g. gnome-panel) depends on.

user@xxx:~$ beagle-info --index-info
Index information:
Name: manpages
Count: 4087
Crawling: False

Name: KOrganizer
Count: 0
Crawling: False

Name: Tomboy
Count: 69
Crawling: False

Name: Thunderbird
Count: 0
Crawling: False

Name: NautilusMetadata
Count: -1
Crawling: False

Name: Labyrinth
Count: 0
Crawling: False

Name: Opera
Count: 0
Crawling: False

Name: Blam
Count: 0
Crawling: False

Name: KAddressBook
Count: 0
Crawling: False

Name: Empathy
Count: 0
Crawling: False

Name: documentation
Count: 37629
Crawling: False

Name: IndexingService
Count: 1311
Crawling: False

Name: Akregator
Count: 0
Crawling: False

Name: KNotes
Count: 0
Crawling: False

Name: Liferea
Count: 0
Crawling: False

Name: applications
Count: 480
Crawling: False

Name: KonquerorHistory
Count: 0
Crawling: False

Name: Files
Count: 136556
Crawling: False

Name: KonqBookmark
Count: 0
Crawling: False

Name: Konversation
Count: 0
Crawling: False

Name: KMail
Count: 0
Crawling: False

Name: Kopete
Count: 0
Crawling: False

Name: Pidgin
Count: 0
Crawling: False

[Jon Eliot (jon-stromme)]

I can confirm this bug, and any Evolution-related indexing is disabled. Tried again after uninstalling Evolution, no change. Retried indexing a number of times, and to start with some (definitely not all) emails were apparently indexed. But after some time, Beagle consistently does not index any Thunderbird email, even when starting from a clean slate with ~/.beagle and /var/cache/beagle removed. xattrs are not enabled on this system.
The directory ~/.beagle/Indexes/ThunderbirdIndex/ToIndex is no longer created (it was created and repeatedly used when incomplete email indexing was done).

Files and Pidgin conversations are indexed (except for a number of file types including .doc where the helper crashes).

This is an AMD64 installation of updated Jaunty.

jon@vora:~$ beagle-info --index-info
Index information:
Name: GMailSearch
Count: -1
Crawling: False

Name: Thunderbird
Count: 0
Crawling: False

Name: Empathy
Count: 0
Crawling: False

Name: Files
Count: 864
Crawling: False

Name: IndexingService
Count: 4
Crawling: False

Name: NautilusMetadata
Count: -1
Crawling: False

Name: Pidgin
Count: 800
Crawling: False

jon@vora:~$

[brott (gatorstudent20)]

I'd like to confirm this. Fresh install of Beagle on Ubuntu Jaunty. Thunderbird is set as a backend, but nothing is being indexed. Also tried running walk_the_dog, but didn't help. Best I could find in the logs is:
grep -n 'thunderbird' *
2009-05-02-15-36-24-Beagle:93:20090502 15:37:48.9717 06541 Beagle DEBUG: Delaying add of file:///etc/thunderbird/profile until FSQ comes across it

beagle-info --index-info
Index information:
Name: Kopete
Count: 0
Crawling: False

Name: Akregator
Count: 0
Crawling: False

Name: KOrganizer
Count: 0
Crawling: False

Name: KonquerorHistory
Count: 23
Crawling: False

Name: Blam
Count: 0
Crawling: False

Name: Konversation
Count: 0
Crawling: False

Name: NautilusMetadata
Count: -1
Crawling: False

Name: Opera
Count: 0
Crawling: False

Name: KAddressBook
Count: 0
Crawling: False

Name: KonqBookmark
Count: 0
Crawling: False

Name: Tomboy
Count: 2
Crawling: False

Name: IndexingService
Count: 0
Crawling: False

Name: Pidgin
Count: 7670
Crawling: False

Name: KNotes
Count: 0
Crawling: False

Name: Labyrinth
Count: 0
Crawling: False

Name: Liferea
Count: 0
Crawling: False

Name: Thunderbird
Count: 0
Crawling: False

Name: Files
Count: 10820
Crawling: False

Name: KMail
Count: 0
Crawling: False

Name: Empathy
Count: 0
Crawling: False

[brott (gatorstudent20)]

Scratch that. I never installed the helper. It's working now.
"sudo apt-get install thunderbird-beagle"

Thanks

[izzy200198 (izzy200198)]

i can confirm this also. beagle 0.3.9 on fresh Jaunty x86 with thunderbird-beagle installed. Beagle indexed less than 10K emails out of a combined 35K emails from two email accounts in thunderbird 2.0.0.21 (meaning two in boxes and two sent folders).

[izzy200198 (izzy200198)]

update: if you go into thunderbirds addons you will see the beagle indexer. from there you can set it to index aggressively, then tell it to drop everything and reset the index. it will re-index everything again fairly quickly. you can then change the indexing speed back to normal. then wait was beagle to complete its indexing process. it may look done in thunderbird, and sits idle. but beagle is not, and you must wait for it to complete indexing the emails you just saw in the thunderbird addon.

for those of you that have not been able to index thunderbird emails or are getting partially indexed emails, please try this step and force it to re-index everything. it worked for me.

[Robert (robrwo)]

I can confirm this bug on Jaunty, AMD64. I note that even though I've installed the thunderbird-beagle package, it does not show up in Thunderbird's add-ons.

[Ron_ (ronald-liebman)]

Izzy, when you say "set it to index aggressively," do you mean Indexing Speed = very fast or very slow?

My TB backend does seem to be indexing my emails, but very little content -- possibly only header information.

[Ron_ (ronald-liebman)]

I've still got a fairly serious problem with Beagle indexing only email headers and not content. I've got Thunderbird 2.0.0.23, Beagle 0.3.9 and Beagle-Indexer (TB Addon) 0.1.3. Please help.

[Ron_ (ronald-liebman)]

I was just informed by one of Beagle's developers that the right place to report these problems is still http://bugzilla.gnome.org/enter_bug.cgi?product=beagle .

[Ron_ (ronald-liebman)]

I've recently tried a couple of things, unsuccessfully, to fix the problem on my own. Make no mistake, this is flailing. I have little understanding of what I am doing, but in the absence of expert support....
1) I downloaded an older copy of the Thunderbird-Beagle add-on, version 0.1.2 . Same problem. Reinstall thunderbird-beagle 0.1.3.
2) I edited the contents of /usr/lib/thunderbird/extensions/{b656ef18-fd76-45e6-95cc-8043f26361e7}/chrome/beagle.jar in the following way:
a) sudo file-roller
b) open the file
c) gedit beagleIndexer.js
d) Based on a discussion I read ( https://bugzilla.gnome.org/show_bug.cgi?id=530632 ), describing a possible fix to the extension for Thunderbird 3 [Note: I am still using TB 2], I replaced 3 instances of "path.unixStyleFilePath" with "nativePath" . Save. Okay to update the jar. Close.
e) This produced Exceptions in the Beagle Log that I did not used to have. Basically, it broke the Beagle backend. Reinstall thunderbird-beagle to undo the changes to beagleIndexer.js .
f) What we learn from this experience is that Beagle's Index Helper process apparently runs Mono to re-open Thunderbird's folders and read the contents. If this is so, the locus of the main problem (i.e., not indexing e-mail content) is the Beagle backend rather than the Thunderbird add-on. That is a body of code that I dare not touch. It's too bad, though, that the add-on does not just dump the e-mail content while it has the chance.

[Ron_ (ronald-liebman)]

I've done a little more digging, in case anyone else cares to jump in. Source code for the Beagle Thunderbird backend is here: http://vbox4.gnome.org/browse/beagle/tree/beagled/ThunderbirdQueryable/ThunderbirdQueryable.cs?h=beagle-tbird-soc07 . The action seems to be around lines 343-347 (called from lines 381-383 and carried forward at line 398.) A partial explanation of what is going on can be found here: http://beagle-project.org/Filter_Tutorial . (See the section "DoOpen() and DoClose().") As I understand it, the file is just a big buffer. Mail messages are laid end-to-end in the file, so the only way to retrieve a particular message is to specify its offset postion and its size. This stream is handed off to parser.ConstructMessage, etc.

I hoping that some reader will have enough experience to spot a potential weakness with respect to Ubuntu 9.X, AMD64, etc.

[Ron_ (ronald-liebman)]

The Beagle community (including Novell) seems unable to offer any further support. Also, I found no evidence that Canonical has changed its support for desktop search apps since introducing Ubuntu 10.04 yesterday. So after reviewing all of the information I could find comparing strengths and weaknesses, I shut down the Beagle daemon and installed Recoll.

So far, I am very pleased. Documentation is good. Installation was a snap. Minor customization (e.g., setting a parm to enable Recoll to work with the Firefox Beagle add-on) was easy. The whole conversion was very uneventful. Creation of the initial index was reasonably fast (under an hour for about 250,000 files.) The GUI is clean and clear. All of the stuff that Beagle caught was there. And all of the stuff that Beagle missed -- like Thunderbird message content -- showed up.

Compare Recoll news...

    * 2010-01-05 : a 1.13.04 is out. It fixes a nasty bug (broken stemming) in 1.13.02.
    * 2010-01-29 : the full Recoll source repository is now hosted on Bitbucket, along with a Wiki and an issues tracking system. Hopefully, this new channel for reporting bugs and make suggestions will increase the feedback rate...
    * 2010-01-05 : a 1.13.02 is out. It brings some nice improvements and new functions. Please try it and report any problems.
    * 2009-12-10 : 1.12.4 is out. It fixes a problem in the preview window search function (qt4 only).
    * 2008-05-22 : we now have a mailing list:
          o Subscription management
          o Archives

... to Beagle news:
21 Jan 2010 Beagle status: Beagle isn't in active development. It is getting some occasional maintenance done by Novell.
26 Jan 2009 Beagle 0.3.9 released.
15 July 2008 Beagle 0.3.8 released.
7 Jun 2008 Added initial support for indexing removable medium.
21 May 2008 Added read-only RDF overlay on Beagle index. This RDF store can handle RDF queries and supports different query formats like SPARQL, N3 etc.
14 May 2008 Kio-beagle ported to KDE4.

The future is Recoll.

[lcn_mustard (lcn-mustard)]

Pitty! Was a good program.
My Pc is using kubuntu lucid.
I have the same issue, only for record, I'll try the recoll.
-
~$ beagle-info --index-info
Index information:
Name: Labyrinth
Count: 0
Crawling: False

Name: NautilusMetadata
Count: -1
Crawling: False

Name: Locate
Count: -1
Crawling: False

Name: GMailSearch
Count: -1
Crawling: False

Name: KOrganizer
Count: 0
Crawling: False

Name: Tomboy
Count: 0
Crawling: False

Name: KonquerorHistory
Count: 0
Crawling: False

Name: Opera
Count: 0
Crawling: False

Name: Pidgin
Count: 0
Crawling: False

Name: KonqBookmark
Count: 0
Crawling: False

Name: KNotes
Count: 0
Crawling: False

Name: Files
Count: 6873
Crawling: False

Name: IndexingService
Count: 0
Crawling: False

Name: KAddressBook
Count: 0
Crawling: False

Name: Empathy
Count: 0
Crawling: False

Name: Akregator
Count: 0
Crawling: False

Name: KMail
Count: 5993
Crawling: False

Name: Blam
Count: 0
Crawling: False

Name: Liferea
Count: 0
Crawling: False

Name: Kopete
Count: 0
Crawling: False

Name: Konversation
Count: 0
Crawling: False

[Thomas Hotz (thotz-deactivatedaccount)]

Beagle development stopped, so I fear this bug won't be fixed any more.