Stemming for French language.

[musically_ut]

Some words from French language are not found in the dictionary because they are not reduced to their root words (eg. dansez, tombons, etc.). Hence, a rudimentary stemming strategy for French language is proposed, in line with the strategy being utilized for English language. The Fix is not complete or exhaustive, but should get something started.

Patch attached.

Thanks!

~
musically_ut

[Mehrdad Momeny (mehrdad-momeny)]

Thanks

[musically_ut (musically-ut)]

The suggested fix is, of course, not perfect, and it will eventually result in more bugs. So here are a few short comings:

1. Mis-spelled words in English might end up having a meaning: e.g. cancez -> cancer: Hence, the set of changes are language specific now. I don't know how languages of dictionaries are being managed.

2. The "e-accent aigu" is NOT matched perfectly. I am not very familiar with Unicode string matching, so I did not implement it the 'proper' way.

If you work around any of these, do tell me. On the first sight, they look fairly straightforward fixes.

Thanks.

~
musically_ut

[Majid Ramezanpour (thinkgnu)]

Actually we are working on a new series of MDic to be more compatible with KDE so we will use KDELibs in it instead of Qt.

In this series we need a better designing of classes and also make it more Plugin based.

One of our plan for stemming strategy is to make it Plugin based, so it will be easier for users to make Plugin for stemming strategies for their own languages.

So if you have any idea about designing it in a better way we highly appreciate it.

[musically_ut (musically-ut)]

I can see a simple way of dealing with simple morphological rules, simple text files as plugins. For example, if one allows perl like regular expression matching (though the current rules can be implemented using only prefix/suffix matching), I would write out the replacement rules as the follows:

english.rep:
---------------------------------------------------------
/ies$/ -> 'y'
/n't$/ -> ''
/ness$/ -> ''
/'s$/ -> ''
/ied$/ -> 'y'
/^un/ -> ''
/ment$/ -> ''
/ing$/ -> '', 'e'
/ning$/ -> 'n'
/s$/ -> ''
/ly$/ -> ''
/ed$/ -> '', 'e'
/es$/ -> '', 'e'
/er$/ -> ''
/ily$/ -> 'y'
---------------------------------------------------------

french.rep
---------------------------------------------------------
/ons$/ -> 'er', 're', 'ir'
/ez$/ -> 'er', 're', 'ir'
/ent$/ -> 'er', 're', 'ir'
/es$/ -> 'er'
/s$/ -> '', 're'
/e$/ -> 'er'
/u$/ -> 're'
/(e-accent ague)$/ -> 'er'
/i$/ -> 'ir'
// -> 're'
---------------------------------------------------------

How does that sound?

~
musically_ut

PS: And, while writing that, I stumbled across another bug, so the corrected diff is attached.
PPS: Unfortunately, as KDE4 had scanty proxy support, I have shifted almost completely to Gnome. :)

[Mehrdad Momeny (mehrdad-momeny)]

It seems fine,
thanks, we need more research on this topic. ;)

Sorry, I was disconnected for 3 days.