Face detection / recognition

[Javier Jardón]

This feature is already present in other photo management software, like iPhoto or Picasa

It would be very cool if some free software get such a feature. Maybe it could be implemented as a general approach (so other project could use it)

[Christian Weiske (cweiske)]

Version: (using KDE KDE 3.5.5)
Installed from: Gentoo Packages

Reaching my personal 10.000 photos barrier in the next weeks, I face a problem:
Tagging photos by hand is not doable anymore since it takes too much time. While setting tags for a whole album (france, wedding, 2007) is easy and fast to do, tagging each of the 200+ photos in an album with the people on it is impossible.

When searching for photos, you often try to find a person - how it looked at a time, or generally need all photos of a certain person. Tags help here, but not all photos are tagged through the reason above. What I would like to see is a plugin that automatically detects faces and allows one to drag a tag from the tag tree on, or just add a tag to the detected face.

There are some face detection and recognition libraries out there that could be used, one well-known is the Intel's OpenCV lib [1]. libFace [2] is another one that specializes in face rec/det and uses intel's lib internally.

Gnome's F-Spot will get this feature trough the summer of code project [3, 4]. If it gets that, it'll be my personal killer feature to use fspot.

It would be very cool if digkam would get such a feature. And by the way, thanks for making digikam. My girlfriend and I use it since years, and it's been unvaluable for managing our photo collection.

References:
1 http://sourceforge.net/projects/opencvlibrary
2 http://libface.sourceforge.net/
3 http://idea.zanestate.edu/archives/2007/04/face-recognition-coming-to-f-spot/
4 http://bugzilla.gnome.org/show_bug.cgi?id=324391

Mailing list thread:
http://mail.kde.org/pipermail/digikam-users/2007-June/003642.html

[Stephan (stephanolbrich)]

*** This bug has been confirmed by popular vote. ***

[Caulier-gilles-9 (caulier-gilles-9)]

Hey, i have never seen a wish confirmed after 2 days than it have been open...

Marcel, what do you think about Face detection. For me it sound a killer feature...

Gilles Caulier

[Marcel-wiesweg (marcel-wiesweg)]

Yes it sounds very interesting.
We would need to find out what the workflow is (training of the face recognition?), and how well the library works.
Then it's about writing an easy to use wrapper class for the library.
And finally integrating it to digikam, perhaps in the image search or in some yet to be written tagging workflow?

[Caulier-gilles-9 (caulier-gilles-9)]

Marcel,

... or during downloading from camera...

Gilles

[Arnd-baecker (arnd-baecker)]

Well, this will be a long road until success:
I had a quick look at installing opencv (which fortunately exists
in debian etch as version 0.9.7-4).
Then libface needs some tweaking in the header files so that opencv is found,
but compiles fine after that. (I can post more details on that if needed).
Then one has to look in the examples directory of libface-0.1,
and compile
constructmat.cpp facedetect.cpp recognize.cpp train.cpp
However, it is not clear from the (essentially lacking, unless
I missed something) documentation how to use these:
Presumably it is train followed by recognize or facedetect.
For train one has to supply a file which contains names and
images of faces to be trained.
So far I only managed to get a segmentation fault...
(note to self: don't use my own pictures, but those of models,
the code seems to be picky ... ;-)

Then there is ellip-mask.jpg which provides a mask, maybe(?) used
for the input images. Whether the algorithm is able to detect
the region of interest with a face to be recognized is not clear to me.
Presumably all this will require some more digging in the original opencv
documentation.

Also note that the libface project does not appear that active,
after an initial check-in 6 weeks ago nothing happened...

OK, I will stop at this point, maybe someone else has more luck ...

[Caulier-gilles-9 (caulier-gilles-9)]

Arnd,

I'm not surprised by this feedback (:=))). I know very well the world of experimental algorithms study (i'm work in a French scientific research center)

With this experience, i'm always aware about it. It's always an instable world witch need a long time to be stabilized.

Also, if the project of this shared library is not really active, we need to wait is something will be done during Google Summer Code to improve this situation.

My viewpoint is this : Face Detection is an interressing feature of course, but we have more important task to do before KDE4/Qt4 or DB update port for ex. (:=)))

So we wait and see...

Gilles

[Arnd-baecker (arnd-baecker)]

> So we wait and see...

Yes, there is a lot more concrete stuff...
Apart from the technical side of the KDE4 transition,
there are also non-trivial workflow aspects
to be addressed wrt tagging (e.g
http://bugs.kde.org/show_bug.cgi?id=114465)
and improving searching/tag-filtering ...

[Christian Weiske (cweiske)]

What I understood so far about opencv:
- There are two things: Face detection and face recognition. The first is to actually detect the position and region of a face, or all the faces on the photo. The second is to recognize the face at the detected region and say who it is. This needs a training set.

The simple / first thing I think should be implemented is face detection and face outline display in digikam. The user OR a program would say where the the faces are (e.g. user could select a frame as done for selection now), and digikam would store this information somewhere in the database. Further, a tag shall be associatable to that selected region (that would not only allow faces, but other details to be selected - as flickr allows it). This way, you can use a person name/tag and tag photo regions.

The second is face recognition. With the things implemented in the first point, it should be possible to extract the faces/regions from already tagged photos and train the face recognitioner. Untagged regions could then be sent to the recognition engine/script and be tagged automatically after.

So in short: Digikam first needs a way to tag/describe regions of an image. I think rectangles should be fine, no need to make it complicated and support circles or even AND OR and whatnot. After this, even outside scripts can be used to facedetect and recognition.

[Arnd-baecker (arnd-baecker)]

My impression is that first the algorithmic side of both steps
- face detection
- face recognition
has to be working first (well, it did not for me in the first try...),
before the integration into digikam can be done.
All can be tested in detail separately
and this should then also clarify the needs on the digikam side.

Your suggestion to describe regions of an image and
associate those with a tag is indeed something of use
independent of the face detection.
This should have many applications (--> separate B.K.O entry?)

Best, Arnd

[Christian Weiske (cweiske)]

Arndt, I don't think so. OpenCV already provides a working algorithm set, so digikam needs the infrastructure to use the results of that algorithms - that is, mark a region of a photo as "face", and to say that this region/face is someone - tag it. Only then detection/recog are useful. You don't need fuel until your car has wheels.

[Arnd-baecker (arnd-baecker)]

So obviously I missed this then - there is a fully working code based on OpenCV
to which you supply images (eg. just containing the face part) to train
and then do the recognition for new images? Do you have a pointer?
But what is the role of libface then? Is it needed? Did you manage to
get it working?

[Christian Weiske (cweiske)]

Arnd, it's my fault. OpenCV is a lib that can be used if one wants to do face detect/recognition, and libface would be the one that could be used by digikam. But if it crashes regularly, it doesn't help. Are there other libs out there that could be used, or even a program via cmdline?

[Christian Weiske (cweiske)]

http://openbio.sourceforge.net/ is pretty old, but it has the cmdline programs

[Colin Guthrie (launchpad-colin)]

In Reply to Comment #9:
> Your suggestion to describe regions of an image and
> associate those with a tag is indeed something of use
> independent of the face detection.
> This should have many applications (--> separate B.K.O entry?)

Yes this would be very good. I think a new BKO is needed. From my point of view this would help for my Sync plugin (yeah I know I've made 0 progress lately!) when integrating with e.g. Facebook - Tags of regions to represent your friends etc. I'd imagine an extension of the current tags concept to store always a region (defaults to the whole image) of an item. Smaller regions can be defined that represent e.g. faces or other special notes of interest.

I can see how Face Detection could automate the assignment of these people tags.

I wonder however if such a capability could be included as a kipi plugin? What more would the API need to do to enable this. We are always discussing when we'll break the API anyway so perhaps the requirements of face detection should help mould the API changes?

Col

[Christian Weiske (cweiske)]

I opened a new bug #146337 for region tagging

[Colin Guthrie (launchpad-colin)]

To clarify: when I said:
> I wonder however if such a capability could be included as
> a kipi plugin? What more would the API need to do to enable
> this. We are always discussing when we'll break the API anyway
> so perhaps the requirements of face detection should help mould
> the API changes?

I was referring to the face detection stuff, rather than the region tagging stuff. Should be obvious if you think about it, but wanted to clarify :)

[Christian Weiske (cweiske)]

Two interesting links for face recognition:
http://www.iis.fraunhofer.de/bf/bv/kognitiv/biom/dd.jsp
(Fraunhofer Institut provides a demo application to detect faces. Perhaps someone could talk to them and convince them to open-source the algorithm? Page is in German)

http://www.riya.com
Is a visual search engine where you can upload your pictures, unfortunately they do not allow/provide offline use.

[Christian Weiske (cweiske)]

It seems the f-spot code is ready:
http://apart-dev.blogspot.com/

[Caulier-gilles-9 (caulier-gilles-9)]

ok, face detection is fun...

Christian, we will be busy on KDE4 for few months (XMP support + new Database), but i'm interressed to take a look later about this code...

So i will be happy to have more users feedback about this code especially statistics to detect the same face on a collection of picture (to apply a tag automaticly)...

Thanks in advance

Gilles

[Arnd-baecker (arnd-baecker)]

Just to add two more links about face-recognition (discussed in the German
magazine c't 13, pp 168, 2008):
http://ivt.sourceforge.net/
http://www.face-rec.org/

Ivt is a GPL library which employs a USB webcam (etc.)
for face detection and face recognition.
Internally it uses OpenCV, however I don't know whether anything
additional is contained which might be useful for our situation with still images.

[Caulier-gilles-9 (caulier-gilles-9)]

Arnd,

Andi, know OpenCV library. he develop a new kipi-plugin to fix automatically red-eyes. He can give more info about if necessary.

Gilles

[Michael Skiba (opensource-michael-skiba)]

Here's a collection of additional links to programs which deals with this topics - note however that this page is dated 2005. But maybe there's something valuable..
http://pages.cpsc.ucalgary.ca/~hanlen/vision/facelinks.html

[S-dem (s-dem)]

Version: (using KDE 3.5.9)
OS: Linux
Installed from: Gentoo Packages

It would be very nice if photo's could be automatically tagged by using face recognition techniques. All your photo's could be scanned for people, then grouped by similarity and tagged all at once. If already tagged people are found, they could be tagged without user intervention.

Would be very nice!

[Jan Kundrát (jktjkt)]

Got a library for face recognition?

[S-dem (s-dem)]

Maybe OpenCV: http://opencv.willowgarage.com/wiki/ ?

[S-dem (s-dem)]

I believe digikam already uses this library to perform automatic red-eye correction...

[Caulier-gilles-9 (caulier-gilles-9)]

This file is definitively for KDE4

Gilles Caulier

[Julien (julien-narboux)]

A page which explains how to use opencv for face detection: http://opencv.willowgarage.com/wiki/FaceDetection

[Andi-clemens (andi-clemens)]

Face Detection is not that hard, and there are ready-to-use classifiers as well in the OpenCV package.
But unfortunately those classifiers are only working very precisely when using for frontal faces. If the head is turned to the left or right or if hair hides some parts of the face, detection fails.

The much harder part is face recognition and I don't know how to make it learn the classifier that fast. As I trained the classifier for eyes (RemoveRedEyes plugin), it took two weeks to extract the features. The classifier isn't even that good, I might need to train another.

I really don't know how Picasa is going to do this.

Andi

[Andi-clemens (andi-clemens)]

(In reply to comment #25)
> I really don't know how Picasa is going to do this.

iPhoto I mean...

[Gaetan Semet (gaetan-xeberon)]

I do agree this could be a great addition to digikam software.
Currently, iPhoto and Picasa (web) are doing this.

- Go or you picasa web album and click on "Person" tab.
- all faces are detected (I suspect a script is run periodically to update the cache).
- you access to a serie of batch of photo of the same person and you just enter his/her name.

I would suggest, for this feature to be as killing as possible:
- First, perform a face detection on each photo in the entier library or a smaller part. Then, sort and display first the photos on the person on the most shots. This is psychologically important and have a strong impact since the user will think "whow! it is able to find all the photo with my mum, whoaa!!!". There is also lot of single photo (faces too different), but displayed last it have less impact and this can be entered latter
- of course if the scan already have been performed, the result should be cached/saved somewhere so we don't have to perform a new scan each time.
- In each batch of photo, ask to enter his/her name and tag the photo accordingly (please, use metadata tag written in the file itself, accordingly to a rule like "Persons/Name of the person")
- When adding photo, automatically performs face detection and display the most relevant name, or allow to enter his/her name

[8pp-u26-gj5 (8pp-u26-gj5)]

This is a real killer feature and I urge including it into digikam.

[Alexander-balzer (alexander-balzer)]

Fspot has it, Picasa has it already, iPhoto has it now.
When digikam is last in implementing users will switch.
I think this is a feature people would switch for!

[Stéphane List (slist)]

Hi,

You can find an example of howto integrate opencv mechanism into a QT application on my new webpage :
http://slist.lilotux.net/linux/qmotion/

I'm currently working on a similar project for face detection and face recognition.

Don't hesitate to get in touch with me for more informations.

Stephane

[Andi-clemens (andi-clemens)]

We are already using OpenCV in one of the KIPI plugins.
Face detection in general should not be the problem, but recognition seems to be hard.
What do you plan to solve this problem?

If you don't want to post here, you can sent my a private mail. I used OpenCV for my diploma thesis, it's been a while but I think at the moment I'm one of the devs with the most experience in OpenCV.

Andi

[Caulier-gilles-9 (caulier-gilles-9)]

To all,

This is my vision for Faces auto tagging...

If you use Facebook, you can mark people face over images. this is simple and great. I think it's the first step to do: added a small to tool to mark face manually in digiKam. For this stage, i know somebody in Mandriva who have work on this subject...

The second step, is to implement a tool to use current faces database done manually to be abble to find automatically similar faces over the whole collection of image. This can be done also during image downloading from camera...

For this second step, OpenCv can be used to parse image contents and detect face. Of course we need another algorthim somewhere to create faces fingerprints and compare faces one by one. For me it's sond similar than Haar stuff, but using more precision and details.

Perhaps, OpenCv provide something like this. Here i don't know.

What do you think about ?

Gilles

[Alexander-balzer (alexander-balzer)]

Maybe somebody can translate this, I found it quite interesting.
http://www.heise.de/tr/Gesichtserkennung-Clever-oder-unheimlich--/artikel/134269/0/0

The topic face detection/recognition is very popular at the moment.

[Sgbotsford (sgbotsford)]

Could this be implemented as a KIPI plugin, thus making it useful for more than just digikam?

For the recognition part, it would help if an album could be given a group of people. If I'm at a family reunion, then I've got a different set of faces than when I take my grade 10 class on a canoe trip. Have a more restricted set of faces will increase the accuracy of the recognition software.

Peopletags is another case of album specific tagging -- I don't want to see my inlaws in the tags on my canoe trip. I don't want to see my grade 10's when working on my family tree.

[Eros-kde (eros-kde)]

I saw this GSoC 2009 proposal:
http://socghop.appspot.com/document/show/user/hoheinzollern/digikam_face_rec

The proposal sounds well formed and the use case is really solid!
I hope this gets selected for the summer of code because Digikam will have a very distinctive feature and will get a lot of hype then! ;-)

Enrico Ros

[Caulier-gilles-9 (caulier-gilles-9)]

But who have posted this subject ? Andi, Marcel, Arnd, Mik ?

It's not me...

Gilles Caulier

[Andi-clemens (andi-clemens)]

No, I have not posted anything...

[Alessandro Bruni (alessandro-bruni)]

Hello, it's me.

I'm a student who made a proposal through the GSoC app. I tried to contact you - Gilles - by mail last week.
If you have any comment and want to mentor me let me know!

Alessandro Bruni

[Caulier-gilles-9 (caulier-gilles-9)]

Alessandro,

I can mentor you, but i'm not a specialist of OpenCV. Andi know very well.

Marcel can help you about all database questions.

I can guide you for the rest.

I'm very overloaded currently. I give evening classes for webmaster in my computer science school, and it take a while to prepare. This is why i'm late to manage my pending mail. Sorry.

Gilles Caulier

[Alessandro Bruni (alessandro-bruni)]

Thank you for the answer Gilles :-)
I just wanted to get some feedback as we're getting to the end of the period of acceptance, so time is precious.
Good work and don't worry.

So! Andi, would you like to mentor me? ;-) If you want you should become a mentor in the socghop.appspot.com application and comment my proposal.

My regards,
Alessandro

[Andi-clemens (andi-clemens)]

(In reply to comment #39)
> I can mentor you, but i'm not a specialist of OpenCV. Andi know very well.

Gilles,

I'm not a specialist in OpenCV, too. I used it once (this is two years ago), but not very indeep. I can tell you how to use the built-in face detection, but you can read about this on a lot of webpages, too. Actually this is just 10 lines of code.

(In reply to comment #40)
> So! Andi, would you like to mentor me? ;-) If you want you should become a
> mentor in the socghop.appspot.com application and comment my proposal.

To be honest, I don't know if I am the right guy for this. I've been with digikam for 8 months now, but still have not done big things with the source code. I'm not so familiar with the code (without the indexer of Eclipse CDT I wouldn't find anything ;-)). I sure could try to assist you with OpenCV, but maybe someone else should be the mentor. By the way: even if you use OpenCV, you have to program the face recognition by yourself, there is no such feature in the library, at least I haven't seen one. So just relying on OpenCV doesn't help here :-) You must find your own ways to recognize faces.

Andi

[Alessandro Bruni (alessandro-bruni)]

I come back again with some -fresh new- old info:
http://gsoc-wiki.osuosl.org/index.php/Finding_mentors
Here it's said that you can both mentor me: maybe Gilles should be the primary mentor for my project and Andi a more technically skilled mentor (for the openCV library).
If you agree we should notice it to the kde team now, as we're reaching the end of the interim period:
http://socghop.appspot.com/document/show/program/google/gsoc2009/timeline

Alessandro

[Alessandro Bruni (alessandro-bruni)]

Also created a new section in kde wiki's GSoC page:

http://techbase.kde.org/Projects/Summer_of_Code/2009/Ideas#Project:_Face_Recognition

[Andi-clemens (andi-clemens)]

(In reply to comment #42)
> Here it's said that you can both mentor me: maybe Gilles should be the primary
> mentor for my project and Andi a more technically skilled mentor (for the
> openCV library).

As I said before, I can try to help you with OpenCV, but I'm also no expert. But at least I own a book :-)

[Alessandro Bruni (alessandro-bruni)]

(In reply to comment #44)
> As I said before, I can try to help you with OpenCV, but I'm also no expert.
> But at least I own a book :-)

Any knowledge of the library is preciuos and good for this project ;-)

That would be a starting point. But I think that the mentor should know better the architecture of the program, so how to act effectively to implement this feature in the best way. For what I saw theese days getting in contact to digikam's developers Gilles is the only one - afaik - who has such knowledge, so I would ask him to be my mentor. But if you, Gilles, think you cannot give me support theese months I would ask Andi to be my mentor. I also talked with Arnd theese days in IRC and I know he has a good knowledge of the project.

Help! I really don't know what to do and rely on your hands for the start of this project. I think this would be a great occasion for me and you, not to mention the end users! That's a killer feature imho!

Alessandro

[Eros-kde (eros-kde)]

Is the project still on hold because of mentor lack? This can't happen! Me and my friends can't wait for the next summer of code to have this feature on Digikam ;-)

Is anybody here registered as a KDE mentor on the SoC 2009 webapp? I'd to vote for this project but I'm not a mentor.. so.. :-/

[Marcel-wiesweg (marcel-wiesweg)]

Well I think Gilles said he can mentor you in comment #39.
This is about the question who is formally listed as a mentor with Google.
I am not registered there, this whole thing passed me by once again while I was abroad the last weeks.

Gilles, Andi and me all have the deep codebase insight that is needed to help get you started, the one more in this area the other in that. You don't get more from anyone else ;-)

I have no problem helping you should you get accepted. (That is true for any new contributor but so few come up and ask code questions :-( )

[Lydia Pintscher (lydia-pintscher)]

Mentor and project selection for KDE for GSoC 2009 has been completed already. Results will not be made public before April 20th. No point in discussion this here further.
Thanks for understanding.

[Caulier-gilles-9 (caulier-gilles-9)]

I would to be clear here about GOsS...

I don't like this. I don't understand what a GOsC mentor need to do exactly, ho to start... etc... And i don't want to read a huge collection of paper from google to only guide a student with Google rules. Sound like very complicated and unadapted to open source project !

To resume my viewpoint : It's a waste of time for me...

I know how to follow students in real job. I do it everyday. I'm also a teacher and i like to share informations, and see working progress in a project.

But seriously, GOsC add rules witch are definitively not adapted to open source project.

In open-source, why to add GOsC constraints ? It's already difficult to follow standard contributors, check/review patches, guide all peoples who want to help. Time is precious and i don't want to lost time with GOsC, to follow students release dates, write papers, etc... with no guaranty of results...

And i have a real life outside this computer.

How i work : when time permit ! Look this entry for ex:

https://bugs.kde.org/show_bug.cgi?id=149485

Here a student, outside GOsC, works on new Liquid Rescale tool for digiKam image editor. And it's work very well. Progressively, the job is done on the right way. There is no advantage for me to use GOsC to follow this guy.

And i work on open-source since a very long time now. I have a very good experience on project management. so, i'm sorry, but GOsC is not an advantage for me...

Gilles Caulier.

[Andi-clemens (andi-clemens)]

I guess the advantage is on the student's side: He can earn a lot of money with it... :-)

Andi

[Kde-2011-08 (kde-2011-08)]

Can this bug be either cleaned up, or refiled? The whole GSoC discussion should have been on the mailing list, not cluttering up bugzilla. With Gilles and Andi's permission I would file a new bug which summarizes the useful comments in this bug with none of the 15 OT comments.

[Caulier-gilles-9 (caulier-gilles-9)]

*** Bug 194401 has been marked as a duplicate of this bug. ***

[Leshiy (alexjironkin)]

Face Detection is easy, OpenCV does this using Haar filter. Face recognition on the other hand is very very very hard. I am pretty positive OpenCV doesn't have this feature, of face recognition, yet.

Although if anybody know of any other non proprietary library that provides that support, would be interesting to have a look at that.

[Leshiy (alexjironkin)]

I had a look around the internet for something suitable, first of all apparently IPhoto 9, I think, has this face recognition technology. But I guess Apple just bought the company that developed it originally, because I just can't find anything about it on the internet.

With regards to open source equivalents, there is libFace, http://libface.sourceforge.net/index.html, but that is in preAlpha and doesn't look it has been active for a while. Other then that, might have to write own code for it. One of the ideas is Eigen Face.

[claudio (claudio-noguera)]

I definitely think this should be implemented. To me there are three steps for developing this:

1. Prepare the database so that it can save where the face is in a picture. It could also be extended to enable region tagging (like facebook or flickr have).
2. Implement face detection.
3. Implement face recognition.

The advantage of this approach is that each of the three steps will be seen as an improvement by the final user. Another advantage is that the whole face detection thing will be easier to implement if it can be divided in incremental steps.

An this is briefly how I think each of the steps could be solved:
1 --> add a "position" field to the imagetags table in the database, it would be empty if it is a global tag.
2 --> train a boosting model based on several haar features. Once trained, digikam should just calculate the haar features and apply the model. This is fast, and notice that no classifier training is done in the computer's machine. Picasa does this. The facedetect example in the openCV library uses too little features, that's why it doesn't work properly (there are too many false positives).
3 --> I think this is a clustering problem, but I have never tried to solve it. I guess it is an easier problem than face detection.

I have some experience in machine learning and computer vision, unfortunately no experience in C++ developing. Anyway, I'd like to help in getting this implemented.

[Leshiy (alexjironkin)]

Are there any good classifiers out there that can be trained on a generic set, and then have good enough accuracy to perform well without any additional training and improvement? Surely the classifier has to be trained on the specific dataset of images first (users database, maybe not the whole, but a selected few images), and then continuasly improved by the user feedback as it is used. Mac's iPhoto has this feature and even with ongoing improvement it still gets the a lot of false positives, that are then weeded out by the user. This way it is not limited by the accuracy of any model that is trained elsewhere.

An obvious case where classifier might perform poorly, is when faces are partialy ocluded by some object. It might recognise it as another person or not recognise it at all. In which case it has to be told "No this is not object A, it is object B". I have also seen that using a number of Haar features can still yield a non face recognised as a face by similarity. So the classifier needs to be told to ignore that somehow.

An additional comment is that, as I understood from description above, the database field caters for a position of a face, not multiple faces. May I suggest to use a list of positions where faces are and what they correspond to. So have a separate classifier table, that has internal representation of people feature set and internal id of a person, which can be then tagged to the face position and id of the person.

E.g. <coordinates of the bounding box (top right and bottom left)>#id of the person: <125,125>,<175,175>#3

Admin question: has this been submited to the GSoC 2009, because I couldn't see it in the project list? Is anyone working on this? Otherwise I will throw my name in the hat to help out with this :)

[Caulier-gilles-9 (caulier-gilles-9)]

>Admin question: has this been submited to the GSoC 2009, because I couldn't see
>it in the project list? Is anyone working on this? Otherwise I will throw my
>name in the hat to help out with this :)

Nobody work on this project for the moment. submitting this subject as GSoC subject is complex and hard to follow. other restrictive subjects have been submitted instead.

Note : personally, I don't like GSoC concept. I don't like to see peoples payed to work on open-source, and followed by mentor not payed as well.

Gilles Caulier

[Caulier-gilles-9 (caulier-gilles-9)]

Alex,

Feel free to work on this subject.

In this room you will see
- Marcel who work on all Database purpose. Talk with him to change DB schema if necessary and to improve DB interface.
- Andi who know OpenCV API and concept (Remove Red Eyes plugin author).

Gilles Caulier

[claudio (claudio-noguera)]

I suggested a db modification in the mailing list that Marcel approved because it was backwards compatible. I think it is suitable for the classifier purpose.

It consists of two new tables: faces and imagefaces.
Faces --> faceid, tagid.
Imagefaces --> imageid, faceid, x1, y1, x2, y2

The faces table is used to match a face with a tag in the database. The other one should be directly written by the classifier. Once you have a trained classifier, it'll first will detect faces in an image. The second step is to assign a different faceid to each different face.

I'd really like to be a part of this, but currently I don't have time.

About the performance of the algorithms, don't be too optimistic. It will be very difficult to recognise ocluded faces. Reducing the number of false positives is easier.

[Leshiy (alexjironkin)]

One of the features that can be used is a Gabor features. I worked with them little bit this year, and they are used quiet a bit in bioinformatics for feature selection. Maybe worth having a look.

K well lets try and get the first part of this done: Getting to face detection to work. This should be fairly straight forward. I'll just take some time to familiarise myself with the digikam code and framework. As a proof of concepr I suggest we should get the Haar face detection from OpenCV working, unless someone has better suggection. I guess that might already be used in red eyes removal, which I have a look shortly.

[Andi-clemens (andi-clemens)]

Don't expect much from this plugin! :) It was my first code in C++, always coded Python before.
Also OpenCV is not really integrated here, and this is a KIPI-plugin, so if we really want to use OpenCV in digiKam, we might want to integrate it there, write a DImg::OpenCvLoader or similar and make it more abstract.

My plugin uses a classifier that I trained for single eyes. The face detection of OpenCV is not that good, it only finds faces that are not covered by other objects. Also the faces need to be straight, not in an angled position, otherwise it fails, too.

[Leshiy (alexjironkin)]

I been snooping around the idea of the boot model. This seems like a really interesting approach. Here I found an implementation of the AdaBoost in C++.

http://codingplayground.blogspot.com/2009/03/adaboost-improve-your-performance.html

[Andi-clemens (andi-clemens)]

AdaBoost is what OpenCV uses for learning classifiers.
But it seems to be very limited, maybe an own implementation would be better here?

Unfortunately I'm not into machine learning that much, so if you have more experience, let the good ideas come :D

[claudio (claudio-noguera)]

I tried myself the opencv facedetect example. I used the python version and made a little python script that applied the tag "person" to every picture with a face. It writes directly into the database, it's not connected with digikam at all.
Two things about this detector:
- Increase the minimum size of a face. I changed it to a minimum of 100x100. That way it takes about 0.6 seconds per picture (my pictures are on average about 7 MPx).
- A lot of false possitives.

Due to the amount of false positives, I think it is worth implementing our own face detection method. However, as a first step, it would be really nice to have this one as a base so we can build from it.

About the opencv approach. It is a cascade classifier built with a boosting technique on a set of haar features. The problem is that the set of haar features is too little, so we'll need a way to specify Haar features and calculate them. To my knowledge, openCV's predefined functions only calculate the reduced set of features used by its classifier.
There are other techniques to detect faces, however this one seems the most appropriate.

[Leshiy (alexjironkin)]

Well from what I understand, AdaBoost is more like a framework to combine multiple weak classifiers. The idea being many weak classifiers, maybe be geared for different parts, like eyes, lips etc, and combination of them is better than a single strong classifier. For AdaBoost the problem becomes of selecting what weak classifiers to use e.g. Haar-like features, maybe something with wavelets, Gabor features, histograms etc.

I guess this is something to keep in mind when plug-in is at that stage.

[Leshiy (alexjironkin)]

I have been thinking about this for a while, and working with OpenCV face detection seems to be quiet good, even if it not 100% accurate.

However, I am still struggling with a way to do face recognition. K a simple way, like using histogram is quiet easy, but what I have in mind is this:

The training of the classifier for the face recognition has to be done continually, e.i. the more it works the better it gets. Learns from itself. But I am struggling to find how that could be done. OpenCV has a haartraining and it would be nice if one could use it to update with the fresh set of the positive and negative data, that has been aquired in between last time the training was run and now. Then use current classifier with this data to produce new one, a better version of the old one. However, I don;t know wheather that is possible in current OpenCV haartraining.

Any other thought on how it could be done?

[Mentalomega (mentalomega)]

Face detection algos are difficult to train. I'd love to have this functioning for KDM as well...

As far as it's development goes, we ARE talking about a trainable system. Might I advise it's 'crowd-sourced'? IE, everyone interested uses it, scans themselves in, then this could be sent back to the devs...it would take a weight off them.

In linking with the rest of the system, could we have it so it works accross mutiple accounts, and could even link to KAddressBook? That would be brilliant beyond doubt...

[Alexander-balzer (alexander-balzer)]

I found something which might be worth a look.

"The Kolmogorov project provides tools for machine perception and tutorials for machine learning, machine perception, and statistics. It supports fast face detection in images and information on Hidden Markov models, Gabor filters, Multivariate Logistic Regression, and Stochastic Differential Equations. It includes MPISearch, a library for detecting objects in images very quickly (real-time in some cases)."

http://freshmeat.net/projects/kolmogorov

[Kde-2011-08 (kde-2011-08)]

> Might
> I advise it's 'crowd-sourced'? IE, everyone interested uses it, scans
> themselves in, then this could be sent back to the devs...it would take a
> weight off them.

I do not see how that would help, but I would gladly submit tagged photos to help.

[Alexander-balzer (alexander-balzer)]

Just found a similar request for kphotoalbum

https://bugs.kde.org/show_bug.cgi?id=170801

[Alexander-balzer (alexander-balzer)]

There is also a feature request for digikam
https://bugs.kde.org/show_bug.cgi?id=146288

Maybe it could be implemented as a general approach (so other project could use it)

[Leshiy (alexjironkin)]

http://freshmeat.net/projects/kolmogorov - Seems to have been moved somewhere.

I had an idea. What if we use the classifier that comes with OpenCV as a base, get the rectangle where the face is. Then use this much smaller image to identify the face. We can use those small images to train classifier in OpenCV. In effect current face detect classifier in OpenCV is as general as it can be, to recognise a generic face, so what we will be creating is a highly specific classifier tailored for each face in the training data, each face in our gallery.

Now, what is needed is to improve general OpenCV classifier locally, because for example, it won't recognise some faces which are altered in spacial domain, e.g. turned 90 deg left/right. I tested this on my machine with an image of Lena (popular test image), which it happely recognises in its original form, but rejects if rotated 90 deg. I had a look at the source code for the haartraining and it needs to be modified slightly because it creates a fresh classifier from scratch everytime, where as what we want is one that takes an existing classifier, then goes through the same motion to update it internally.

Advantages of using OpenCV, as a start, is because it already has lots of features in its bank. However, if that doesn't work well then we can think of an alternative way. I hope to get my hands on some image processing books and papers once I am back at university from holidays. :)

[Julien (julien-narboux)]

Another option which could be considered to have facial recognition in Digikam could be to use the Polar Rose web service (http://www.polarrose.com/).

I have just noticed that there is a module to perform facial recognition using Polar Rose inside Gallery 3 open source gallery system (http://gallery.menalto.com/).

A discussion about this module and how it works is here :
http://gallery.menalto.com/node/86320

The module is here :
http://github.com/gallery/gallery3-contrib/tree/master/modules/polar_rose

It seems that the Polar Rose service can download a rss feed of pictures and return facial recognition information for it.

Of course this raises concerns because pictures have to be uploaded to a third party...

Julien

[Caulier-gilles-9 (caulier-gilles-9)]

If Polar Rose algorithm and method is open-source, perhaps we can back-port code to digiKam core as well and implement something around to interface with database. It will be faster an it will run off line.

Where is the code exactly ? Which language is used to implement Polar Rose tool ?
Do you know a paper which explain how it's work ? Do you a review of tool ? Do you know a comparison with others method ?

Gilles Caulier

[Julien (julien-narboux)]

Sorry Gilles,

The Polar Tool is not open source, it is a closed web service.

Julien

[Alexander-balzer (alexander-balzer)]

The offline version of Picasa now also supports face detection and recognition.
Maybe it helps to get an idea how the gui could look like.

http://www.heise.de/newsticker/Picasa-erkennt-Gesichter-und-schreibt-Geotags--/meldung/145774

[Mfpl (mfpl-deactivatedaccount)]

SVN commit 1048077 by mghansen:

Document for discussing ideas for image annotation and face recognition.

CCBUG: 146288
CCBUG: 146337

 AM ImageAnnotation.odt

WebSVN link: http://websvn.kde.org/?view=rev&revision=1048077

[Sean Reifschneider (jafo)]

You may want to point to OpenCV as an engine for doing face detection and
recognition:

   http://opencv.willowgarage.com/wiki/

Sean

[Alexander-balzer (alexander-balzer)]

Just look at the video. I think this is a very good approach of a face detection / recognition implementation:

http://video.golem.de/details/2413/picasa-3.5.html

[Leshiy (alexjironkin)]

Hi there,

I have been working on getting this to work for the project for the last few days.

There are 2 well established ways of doing it (or so it seems anyway). So I have found them on MATLAB website and now trying to port the code and framework in general to C++. So I thought I share how far I got.

I have went with Fisher's face framework first as it seemed more robust then Eigenface (less dependent on light, pose etc. but not perfect still). However, the problem that I have encountered is this:
    It needs QZ algorithm to solve generalised eigen value problem to work. MATLAB has an implementation of it, but I couldn't find an implementation of it for C++ or anything else other than maths software. It's complex and I don't want to spend time on it just yet if I don't have to.

Because of this I have abandoned this approach for Eigenface, at least to have one working no matter how good or bad it is. So I guess I'll get working on that in next few days and see how far i'll get, hopefully will report with more success next time :)

Alex

[Leshiy (alexjironkin)]

Merry Christmas everyone.

I think this is an appropriate time to report some success in the project :)

I got v0.01 of my Eigenface Recognition to work and recognise different faces, not recognise others and so on. I have tried it on a very limited set of training data, it worked pretty fast, (there are plenty more optimisations to do anyways), but it works :D

I guess this is the first step on a long road of getting it into production and release. I am going to make a bigger training set in next couple of days (whilst digesting all the left over turkey), if it still works, then I will add more "user friendly" features and would like for others to try out, hopefully break it, so we can make it better. At the moment a lot of things like size of the images and so on are fixed and framework doesn't resize them yet, but all in good time.

Alex

[Caulier-gilles-9 (caulier-gilles-9)]

Great to read...

How do you plan to integrate it to digiKam ?

Do you want to make something internal in a first time ? I recommend it to hack indeep without to be disturbed with external depencies.

In the future, do you want to create an external library ? For exemple in kdegraphics/libs, like we do with libkdcraw/libkexiv2/libkipi

How your framework can be used ?
Which data are generated ?
Do you have fingerprints to host in database ?
How do you link a face fingerprint with people name ?
How many time your framework need to recognize a face in a image ?
Can you recognize more than one face at the same time for one image ?

My tips : I recommend to create a dedicated Database file for that to not change current main digiKam DB schema and not bloat main digiKam database file

All best.

Gilles Caulier

[Leshiy (alexjironkin)]

Hi,

I designed it as an external library, where raw image data, or i guess library path can be passed referencing training data. The principle behind it i this:

 - Normalise training data
 - generate eigenvectors and values for a covariance matrix of the data
 - discard some values that are too low,
 - project training data onto 'facespace' (This is essentially PCA)

So in order to see where input image is, it reshapes it, normalises it and multiplies with pre calculated eigenfaces matrix. Then generates Euclidean distance to see where it is closest to. (This is subject to some values which need to be determined by trial and error, which is bit sucky). So there is no person fingerprint per se, as it is combined into signle matrix for all people, but each column in of the projected image matrix uniquely identifies a person. So all you need is table that matches person name with this ID. Then you need to store mean vector and eigenfaces as well as projected images. As you don't want to be computing them everytime. An interesting bit is how to update the matrix, I was thinking taking a mean image across all images for a person, but I don't know how good it is in practice. An additional feature in digikam itself can be to identify a face manually by drawing a box around it, then use it to add or check with face recognition. This is the nature of all classifier, that they won't pick up certain things (I just been using Apple's iPhoto and that's what they use as well).

Face detection is fairly straight forward using haarfeatures in opencv. This classifier has been trained extensively and you can't really do better, without reinventing a wheel. So work flow looks something like this:

digikam library -> use haarfeatures to detect faces on image -> use face recognition on face identified by classifier -> output to digikam to confirm

Then update the Eigenfaces, mean vector and Projected matrix to be reused. Hopefully update makes it better as more data used.

The whole thing works with a single face as input for a person, but obviously more variation you have better it should be.

With regards to databases, apart from having a table for linking coloumn in Projected matrix with a person, I don't see a need to store matrices in it. They are constantly reused and changed, plus they are going to be OpenCV classes, so serialising them backward and forward is unnecessary. I would propose to use a flat file in the background, this is hopefully, doesn't add any more complexity to digikam itself and the whole thing is self contained.

With regards to face recognition, I don't know how familiar you are with haar classifier in opencv, but its good. It does recognise multiple faces at the same time, its not perfect, but nothing is. I think it has been trained on all open to access image databases. Alternatively you can train your own classifier using the same principle, but there is no easy, straight forward way to update it. I have previously toyed with idea of having a cascade classifier for each person, but there pretty much is no way of updating it once it is trained, unless you retrain the whole thing again.

Hopefully this answers at least some of ...

[Caulier-gilles-9 (caulier-gilles-9)]

Alex, Your comment is fine for me.

My tips :

- make re-entrant code in your library. We will use it in a separate thread.
- in batch mode, we will need a progress indicator. It's very important for end users GUI.
- give library version to make detection with digiKam cmake script easy. We have experience with external digiKam libs

If you have some code available to test/review let's me hear...

Do you want to host your code in KDE repository ? (as kdegraphics/libs)

Marcel,

Do you have any questions relevant of face detection database integration ?

Gilles Caulier

[Leshiy (alexjironkin)]

Hi Gilles,

Thanks for the advice and I'll keep all this in mind, but I think this is still early days. There is a lot more testing with bigger libraries to be done and see how it responds to real world images, lots of tweaking to do. I think i will definitely be taking up this offer of more testing from you :)

Once more thorough testing is done, I will try to move the code to KDE repository, so that people can get it easier. I don't believe in making early development code public, as it will inevitable change dramatically very soon, so I will just end up annoying people.

Anyways, hopefully next post will include some results from testing with different images and perhaps some instruction on how to get hold of this.

P.S. If there is anyone who is working on similar system now, let me know so we can perhaps unite to combine the efforts.

Alex

[Johannes Wienke (languitar)]

Maybe it would be a nice idea to put the code into a personal branch of the kde repository right now. This way we can already have a look at it but it wont hurt or annoy anyone. Who will ever depend on code in a private working branch? ;)

[Leshiy (alexjironkin)]

A quick update on the status of things. I finally got KDE SVN account but i can't check anything out with my username :(, but working on it.

Trying to get some integration with digikam now. I am not sure where to put it in though. I have looked through the code and if I understand it right, then I want to put it some where before the image preview is displayed but after it is loaded. Are all images after they have been loaded with custom loader converted to QImage and then displayed?

Alex

[Leshiy (alexjironkin)]

Also a quick question, does digikam use some QCanvas to display images?

The reason is that to annotate image with tags, one doesn't want to draw rectangles and so o on the image itself, but have another layer of widgets on top, for example for face recognition draw a box around and have a label below, which can be edited etc. on a layer above. I don't know if canvas has an ability to do this, unless it can be done differently.

The idea is somewhat similar to panes in Java:
http://java.sun.com/docs/books/tutorial/uiswing/components/rootpane.html

Alex

[Marcel-wiesweg (marcel-wiesweg)]

Alex, we dont have the infrastructure yet you are talking about here.
Michael has written a proposal how to store tags on image regions, but that is not yet implemented. We also dont have any UI, not at all.

We currently do not use a canvas like QGraphicsView. There is related work to do that in the image editor.
We also dont use QImage, but a custom image container...

What part exactly do you intend to test? Is the workflow fully implemented?
> digikam library -> use haarfeatures to detect faces on image -> use face
> recognition on face identified by classifier -> output to digikam to confirm

I imagine there must be a widget to show boxes (autodetected) and to allow to add boxes (manually), then associate a box with a tag (with a tag, a person, and a face identifier).
Should this for starting be done in a separate window? It would be easier then to play with the UI.

Where and how do you store the results of learning?

[Leshiy (alexjironkin)]

I have left the storing of the data to the people that want to use it as far as the library is concerned. In this case, I think it would be appropriate to store in some flat file somewhere, is it is constantly used and changed. Purely because some concerns over storing it in database (see above). The data itself are matrices and vectors of doubles.

Why I have brought it up, pretty much any of the face recognition has to be supervised (decided by a person whether it is match or not), that is why there is a need for a feedback in the process.

I am going to write a small example where library is used and hopefully outputted for feedback. I think then the overall process will become a lot more clear.

P.S. I think I will try and merge with the existing libface library, last I have looked at it on sourceforge, it was unstable and not maintained anymore for a couple of years. This was the reason why I went to write one in the first place.

Alex

[Caulier-gilles-9 (caulier-gilles-9)]

Created attachment 39728
people tag widget screenshot.

At this place from KDE repository:

http://websvn.kde.org/trunk/playground/base/nepomuk-kde/peopletag/

You will find nepomuk people tags widget. This is exactly what's you need as widget, we don't care about the rest of nepomuk to implement face detection GUI.

Gilles Caulier

[Nicolas Réau (kolia)]

@Javier:

Some people have started working on it: https://bugs.kde.org/show_bug.cgi?id=146288 (related to Digikam)

[Mathieu Comandon (strycore)]

Thank you for taking the time to make Ubuntu better. Since what you submitted is not really a bug, or a problem, but rather an idea to improve Ubuntu, you are invited to post your idea in Ubuntu Brainstorm at http://brainstorm.ubuntu.com/ where it can be discussed, voted by the community and reviewed by developers. Thanks for taking the time to share your opinion!

[Javier Jardón (jjardon)]

Hello Mathieu,

Maybe this is not a bug for Ubuntu packages, but It's already a bug for upstream projects, like digikam or F-Spot.
They also have some work done related to this.
See the upstream bugs.

Best regards

[Nicolas Réau (kolia)]

@Javier: This is not a bug neither upstream. if you check the report I was talking about you will see:

Severity: wishlist

This is a popular wish, and indeed people are working on it, but this is not a bug since it doesn't exist yet :)

[Javier Jardón (jjardon)]

Nicolas,

Yeah, It's a bug with wishlist severity ;)

[Leshiy (alexjironkin)]

UPDATE:

I was looking at peopletag project but had problem with it compiling altogether.
I am going to try and get in touch with the developer to add to the people tagger ability to recognise faces through the libface library.

I also will be merging what there is to the libface project on sourceforge, as I think it is the most appropriate place for it.

Progress has been slow as I have a little too much work on my hands lately :( Sorry

Alex

[Caulier-gilles-9 (caulier-gilles-9)]

ALex,

If you need help...

Gilles Caulier

[Alexander-balzer (alexander-balzer)]

@Alex
I have tags of 250 people in ~ 5000 images in digikam. So if you need somebody to test your work. Feel free to contact me.

[7-jm-b (7-jm-b)]

Have you thought about tag storage format ? I have not read any mention of that in this thread, so I suggest using the Fotonotes XML format [1] which is as close as a common image location tagging format as we have. Flickr implementing a derivative of Photonotes [2] is endorsement enough that Fotonotes has the potential.

[1] http://www.fotonotes.net/fotonotexml.html
[2] http://www.fotonotes.net/#installations

The best thing place to store that data would be in the JPEG header. That way, the embedded location tagging metadata would follow the image wherever it goes, whatever the tool used. I can very well imagine the Gallery Tagface module using that information.

[Mfpl (mfpl-deactivatedaccount)]

(In reply to comment #92)
> The best thing place to store that data would be in the JPEG header.

Are there any default places defined for other image formats like PNG, TIFF, etc?

Michael

[7-jm-b (7-jm-b)]

(In reply to comment #93)
> Are there any default places defined for other image formats like PNG, TIFF,
> etc?

A great starting point is the part 3 of the XMP Specifications - "Storage in Files" : http://www.adobe.com/devnet/xmp/pdfs/XMPSpecificationPart3.pdf pages 15 to 25. It provides information about how serialized XMP metadata is packaged into XMP Packets and embedded in different file formats.

Like Fotonotes, XMP is formatted with XML so embedding it into image files is exactly the same problem. But no Fotonotes library is in sight.

Sorry for the lack of proofreading in my preceding message - I shouldn't have posted while I was in a hurry to leave...

[Aditya Bhatt (adityabhatt1991)]

Hi Alex,
I've been playing with openCV for a few months now. I'm quite familiar with Qt too. I'd like to help and get involved with digikam. I'd like to work on the face recognition thingy. I'm still new to bugzilla and svn, but I'll learn soon enough.

[1] My opinion is that the haar cascades bundled with openCV seem quite good, and using multiple types of cascades for different poses might be a good idea. I think one can call the face detector function with very permissive arguments so that almost all faces and some nonfaces are detected, and then apply some of one's own tricks to discard the nonfaces.

[2] Another thing I'd like to mention is - a guy named Rohan Anil has made a PAM face authentication module in the previous two GSOC's for KDE. I've checked his app and the detection part is quite good. In fact, close to perfect.

The catch is that Rohan's app is supposed to detect only ONE face in an image. I think that the previous thing I mentioned in [1] can be used to grab anything that looks remotely like a face and feed it to [2].

His face recognition is something I haven't tested much, but I think it would be, well, *security-level* stuff.

The PAM Module's link is here : http://code.google.com/p/pam-face-authentication/

My bitbucket page : http://bitbucket.org/aditya_bhatt/

Please contact me at : adityabhatt1991 at gmail dot com

Regards,
Aditya

[Leshiy (alexjironkin)]

@Aditya,
I had a look at the "paper" they have published recently and the module itself and not particularly convinced. For example, the system doesn't learn, which means it is limited to the original raining data. As I understand it, this modules uses eyes for face tracking and authentication. There is no guarantee what the eyes will be like in a photo, e.g. an eye is covered tilted rotated and occluded.

The module has high rejection rate for security reasons, and that is something is opposite to what we want here. In practical aspects it builds dxd matrix somewhere inside where d is number of pixels in an image. Take an average 7M pixels image and make that a square. What you get is something unfeasable. Or at the very least needs a lot to optimise. Where as working off a 320x220 image from webcam is a different story. It makes no mention on the limitations of the system, rotation in both x-y plane and z axis is not mentioned as well as the image intensity invariance. What is the limit on how big a face it can detect? In the demo they have on website it only detected image when tester moved closer.

Based on what I have just said and more I am very skeptical about this module. However, having said this I think there are some worth while ideas there like MACE filter and Local Binary Pattern. Perhaps interesting things can also be done with eye detection.

I don't think there is a single system that will do the trick, but instead a number of systems for face detection and recognition put together, working together to reinforce one another will be the best thing in the end.

I don't want to discourage you from trying this approach, so if you write the class based on what they have used, one for face detection and/or one for face recognition feel free to send it to me and I will try to integrate into the library.

By the way since the last post I managed to upload the library to libface (on sourceforge) and this is where it is going to be hosted permanently now. Feel free to download the library and compile it, but be warned the work is still very much in progress :)

Alex

[rohananil (rohan-anil)]

I dont think MACE Filter would be the way to go. The module i am working on is pam-face-authentication is for authentication that is 1 vs 1 matching. For the design - i had to choose between security vs speed (real-time) vs usability. So adapting it for recognition is out of question. So I dont think its fair to compare it for use in recognition :). And question about face detection - its an implementation available on OpenCV based on a method by Viola and Jones. If you want the performance numbers - http://www.lienhart.de/. He has a paper on it. Also again pam-face-authentication is for realtime authentication - So you have to provide your frontal face to the webcam! :) . I have bigger plans for the authentication system once i perfect my AAM implementation http://code.google.com/p/aam-opencv/, Currently the issue is some manual supervision is required which is a NO-GO with a normal user. It just has to work out of the box. Although the results from AAM-Tracking to Recognition is just phenomenal.

Now coming to the task at hand. We are looking at the topic of unconstrained face recognition. Best - research for our photo tagging - could be to implement the best performing algorithm from http://vis-www.cs.umass.edu/lfw/ results. We could go with a support vector machine implementation with highly efficient features. But bottle-neck is the time required for training. Can we compromise over training time ? The more pictures you have more training time. Now we should pickup an algorithm which does not require retraining over the entire set for new images. I think i can code this over this/next weekend. But need help with integrating with the code base. If anyone could provide me an api spec of how the class should be like I would like a go at it..

regards
Rohan

[Leshiy (alexjironkin)]

I really like the idea of http://vis-www.cs.umass.edu/lfw/ Has a lot of useful information, including ROC curves. :)

I am going to create a mailing list for libface so we can discuss the finer points of face recognition and detection there as opposed to boring people here and leave this thread for digikam/Face recognition issues :)

Alex

[Aditya Bhatt (adityabhatt1991)]

Hi,

I'm Aditya, a 2nd year student doing my Bachelors in Information and Communication Tech.
My favorite fields of work are pattern recognition, image processing and computational math.
I'm mostly self-taught in image processing, and also have some knowledge of neural networks. I'm familiar with PCA for dimensionality reduction.

I'm familiar with svn, git and hg version control.
I have some familiarity with clustering algorithms and eigenfaces. Fisherfaces, I'm learning.

I know how to work with Qt fairly well. I also have a very basic knowledge of KDE libraries - which I hope to improve as soon as my exams (a week later) end.

I've been working with Alex Jironkin on his libface project on sourceforge for a while now, and would like to continue working on it over the summer as part of GSoC 2010. I also want to start digiKam hacking soon to integrate this for tagging as described in the GSoC ideas page. I'm already working on a people tagging widget of my own, although I plan to scrap it and start working on the widget in KDE's svn repo as soon as svn:// port gets unblocked in my college :'(. ( I currently commit to libface over https:// )

I'd love it if Alex can be my mentor. And I'd also like the mentorship of one of the digiKam core devs for integrating this into the digiKam interface, and it's tagging database. It'd be nice to get this merged into digiKam before the end of the summer, and I'd like to work on digiKam too in the future - so I consider this a stepping stone.

Post-GSoC, I'd like to work on KIPI and digiKam's "fuzzy" algorithms, as that is my field of interest.

I look forward to your replies soon :)

Regards,
Aditya

[Kde-2011-08 (kde-2011-08)]

It seems that KDE did not get a spot in GSOC this year:
http://socghop.appspot.com/gsoc/program/accepted_orgs/google/gsoc2010

[Arnd-baecker (arnd-baecker)]

> It seems that KDE did not get a spot in GSOC this year:
> http://socghop.appspot.com/gsoc/program/accepted_orgs/google/gsoc2010

Sure? I see KDE listed there (in the second part).

[Leshiy (alexjironkin)]

I was going to say, that it would be very suprising if they didn't. I can see them in organisations completed profile lower down the list too.

[Kde-2011-08 (kde-2011-08)]

> (in the second part)

Second part?!? Who gets that far?

Sorry for the mistake, I'll go away quietly now...

[Nareshsankapelly (nareshsankapelly)]

hi,

I am a student pursuing B.Tech 4th year from Indian Institute of Information Technology- Allahabad, India.

I am keen to get involved in open source project development. I have seen on KDE website the project ideas for GSoC 2010. I would like to work on Face Recognition Project. I took this project because I have worked on the hand gesture recognition using OpenCV and am currently working on the same area.You can have a look at my work here.

http://www.youtube.com/watch?v=wnKk07-YNFw&feature=player_embedded

I am having good knowledge of C++, Qt and the classification algorithms.

I would like to know some more details of the project.

I would be happy to know answers to these following questions.
1. what classification algorithm is to be refined for face detection part. OpenCV provides Haar Classifier using Ada boost learning and it works with almost 99% accuracy. Is it the same algorithm that is to be refined or something else?
2. If it is Haar classifier to be refined. In which part it is to be refined haartraining utility or haar detection part ?

Hoping for a positive reply from you as soon as possible.

Thanks
Naresh Reddy S.
IIIT-Allahabad

[Leshiy (alexjironkin)]

Haar training is not something of interest, because it doesn't provide ability to update on the fly. Unless you can think of a way that can be done because haartraining learns from scratch everytime. So you need to write updater from scratch and do some clever hacking of retrieving previous states from learning.

So we are interested in refining face detection first. When you say it is 99% good, it also has high rate of Type II errors, identifying something as a face when its not, which is bad. That is what needs refining. This problem arises when you use real photos, because need to specify feature size in the identification.

Or you can think of a better face detection mechanism

Alex

[Leshiy (alexjironkin)]

I seem to be keep shooting people down with ideas. :( Sorry.

I think we got face detection to work fine now, the area where we need help and fresh ideas is face recognition. So if you have idea or even better an algorithm for face recognition feel free to post it to me or into libface mailing list and we will get on it to be implemented :)

I guess another area would be deeper integration into KDE itself, like with contacts, I guess this is also can be done in digikam.

Alex

[Johannes Wienke (languitar)]

At my university we use active appearance models to recognize faces. This approach needs a lot of training data to set up good models but should then scale much better then eg. Eigenfaces. Unfortunately I can only find the diploma thesis about this which is in German:
http://www.airesearch.techfak.uni-bielefeld.de/files/Lang2007-DIP.pdf

[Johannes Wienke (languitar)]

And a few more links:
http://www.springerlink.com/content/tyy00w31extww8p1/
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V09-3Y4510K-1B&_user=10&_coverDate=06%2F30%2F1995&_rdoc=1&_fmt=high&_orig=search&_sort=d&_docanchor=&view=c&_searchStrId=1263077754&_rerunOrigin=scholar.google&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=c9834a5a69604f23126bf53e06e6655c

[Leshiy (alexjironkin)]

Does the whole thing needs to be retrained everytime you want to add more data or can it just be updated.

One of the problems with using Haar features for face recognition now, is because you also need a large training set, and you can't easily add more data to the cascade, in other words it has to be retrained from scratch everytime, which is a big problem.

Also bear in mind this is supervised learning, so there is a reliable input from people to tell what is right and wrong. So only correct data is added to the classifier/cascade.

Thanks for the idea though, would be interesting to have a look at that at some point.

Alex

[Johannes Wienke (languitar)]

This depends on the classifier you want use. AAM only provides the feature vector. The solution Christian used was a two step procedure:
1. Nearest Neighbour for rejection (trivial update ;) )
2. Classification based on Gaussian Mixture Models. There will certainly exists enough algorithms to update them with new samples.

[Leshiy (alexjironkin)]

    Accepted projects for GSOC 2010 were announced last night. I am also happy to say that one of the projects accepted was face recognition project. :)

    Adytia will be doing this project and hopefuly we will have a fully working plugin by the en of the summer.

Alex

[Caulier-gilles-9 (caulier-gilles-9)]

Great that it have been accepted. I'm very impatient to test this feature.

I talk by IRC with Adytia this week end about face detection, and making a plugin is not the good way. It's better to patch digiKam as well, to be able to work directly with digiKam DB. I will attached the conversation to this file, for details.

I plan to organize a coding sprint for digiKam at end of summer in France, Aix en provence. It wil be nice to see libface developpers join us at this event. Note : KDE-ev will sponsorship the event (travel + hotel payed).

Gilles Caulier

[Caulier-gilles-9 (caulier-gilles-9)]

digiKam IRC talk between me and Aditya Bhatt about face detection.

[19:55] <cauliergilles> Adityab: ping ?
[19:55] <Adityab> cauliergilles: Yes?
[19:56] <cauliergilles> Adityab: hi. Just 2 questions
[19:56] <Adityab> cauliergilles: sure
[19:56] <cauliergilles> Adityab: 1. Do you recieve your svn account as
developper ?
[19:57] <cauliergilles> Adityab: 2. in digikam-devel mailing list, do
you recieve my mail about digiKam database problem today (i'm not sure
if this message is passed right because it's long)
[19:57] <Adityab> cauliergilles: No, I'm applying tomorrow. They give
higher priority to GSoC students. And even if I don't get into GSoC,
I'm still applying.
[19:57] <cauliergilles> Adityab: ok. let's me hear if you have some pb
about. I can contact right admin accordingly
[19:58] <cauliergilles> Adityab: and don't forget to add you in coding
sprint oo.org document from svn.
[19:59] <cauliergilles> Adityab: I will need quickly all info from
contributors to book hotel and restaurant at Aix en Provence
[19:59] <Adityab> cauliergilles: Thank you :) Yes, I received the mail
and I read it, but I didn't try to analyse everything as my final
exams are from tomorrow
[20:00] <Adityab> cauliergilles: Yes, I'll update the document soon.
I'll be renewing my passport also, and that will take a max of one
month.
[20:01] <cauliergilles> Adityab: ok. no problem. I just want to be
sure if mail is posted right through KDE server. Normally Marcel must
respond about
[20:01] <cauliergilles> Adityab: great.
[20:01] <cauliergilles> Adityab: currently, i don't share a lots of
time with GoSC, because i change my home, and i very busy.
[20:02] <Adityab> cauliergilles: That's okay :)
[20:02] <cauliergilles> Adityab: i also, contribute a lots in Exiv2
project to finalize Sony Makernotes support.
[20:02] <Adityab> cauliergilles: Thanks :) If you can act as my
referrer to the sysadmins, I'll apply just now.
[20:02] <cauliergilles> Adityab: i'm also teacher at Aix en provence,
in a computer science school
[20:03] <cauliergilles> Adityab: so my free time is limited... (:=)))
[20:03] <Adityab> cauliergilles: Must be a hectic job :)
[20:03] <cauliergilles> Adityab: but i will switch indeep to digiKam
and follow GoSC more seriously in few week
[20:04] <cauliergilles> Adityab: i'm sure that Marcel will follow you
properly. He is serious developer
[20:04] <Adityab> cauliergilles: Yes, he is very helpful. He replied
to my every query
[20:04] <cauliergilles> Adityab: Anyway, if you have question or a
problem don't hesitate to mail me.
[20:05] <cauliergilles> Adityab: i'm not online frequently in this
channel, due my external job
[20:05] <cauliergilles> Adityab: but i read my mail each day
[20:05] <Adityab> cauliergilles: Sure, thanks :)
[20:06] <cauliergilles> Adityab: yesterday, i do a presentation of
digiKam in a LUG at Aix en provence
[20:07] <Adityab> Yes, I saw your update on facebook :)
[20:07] <cauliergilles> Adityab: ah yes.
[20:08] <cauliergilles> Adityab: and a lots of people ask me if face
detection will be implemented soon, to provide the same feature than
Picasa
[20:08] <cauliergilles> Adityab: i said ... yes of course
[20:08] <cauliergilles> Adityab: so people a...

[Caulier-gilles-9 (caulier-gilles-9)]

Implemented in digiKam 2.0.0

Gilles Caulier

[Simon-44ecklpsr (simon-44ecklpsr)]

yay, party on!

[Andreas Brauchli (blk)]

fix upstream bug duplicate

[Guillaume Hain (zedtux)]

I already give it a try with my QuiEstLa project: https://launchpad.net/quiestla.

I did it for Shotwell in this way: Provide an OpenCV client embedded into a D-Bus application.
My project is working, and need some review / optimizations.

It's a client/server application.
I already described how to start it and use the client application to test it.

I'm open for pull requests / ideas / suggestions / code reviews.

[Isilmendil (isilmendil)]

*** This bug has been marked as a duplicate of bug 292900 ***