Import problem log incomplete

[Michael Hendry]

Binary package hint: shotwell

At the end of the import process I got the following report:

149 duplicate photos were not imported:
/media/Elements/Pictures/Coolscan2/2001-19/IMG012 2900.jpg
/media/Elements/Pictures/Coolscan2/2001-05/IMG014 2900.jpg
/media/Elements/Pictures/Coolscan2/1977-13/IMG021 2900.jpg
/media/Elements/Pictures/Image Library One/My Images/113-1367_img.jpg
(and 145 more)

1 photo failed to import due to a file or hardware error:
/media/Elements/Pictures/Image Library One/My Images/2005_10_01/IMG_1403.JPG

182 unsupported photos skipped:
/media/Elements/Pictures/Coolscan2/1972-08-27/Image12.nef
/media/Elements/Pictures/Coolscan2/1972-08-27/Image13.nef
/media/Elements/Pictures/Coolscan2/1972-08-27/Image14.nef
/media/Elements/Pictures/Coolscan2/1972-08-27/Image15.nef
(and 178 more)

21611 photos successfully imported.

I'd like to be able to trace all of the files that Shotwell had problems with, so that I can ensure that Shotwell can see them in future.

[Michael Hendry (hendry-michael)]

Further investigation of the 149 duplicates photos:

I used Shotwell to import everything in my /media/Elements/Pictures directory, and got the truncated error report above.

I've now used the File Manager to search the same directory (and sub-directories) for "IMG012 2900.jpg", and although there are 60 files with this name there is no duplication of the image "/media/Elements/Pictures/Coolscan2/2001-19/IMG012 2900.jpg".

What criteria are used to establish that a given image file is a duplicate of another? It can't be filename, because there would be far more than 145 duplicates.

I'm also puzzled that a group of NEF files is marked "unsupported". These NEF files were produced by a Nikon product (Coolscan IV), so presumably they conform with Nikon's rules for NEF files.

[Jim Nelson (yorba-jim)]

Filenames are not used to detect duplicates with Shotwell. (Obviously if it's the same file *path* it's a duplicate, but that's a special case.) We do an MD5 hash of the file's contents to detect duplicates. In other words, if those files have the same file content as one already imported (as though they were copied), then Shotwell will not import them.

Note that we have an outstanding bug at http://trac.yorba.org/ticket/2587 that deals with our metadata library returning the same bogus thumbnail from two different files, generating a false positive. This is because we use an MD5 hash of the embedded thumbnail for duplicate detection as well. As that ticket discusses, we might move away from this, as it's not especially reliable.

So, my questions are: (a) Are these files in fact byte-for-byte duplicates, or (b) do they have the same embedded thumbnail, or (c) are they victims of this bug as well.

If you know they're (a), then that's designed behavior. If they're not, you can send two of them to me and I'll examine them and see what's triggering this behavior.

Regarding the NEF problem, that format should be supported by Shotwell. (I have a handful of them in my test library.) We rely on libraw (which is based off of dcraw), so it may be that these particular NEF files gave the RAW library decode problems. If you send me one of these files, I can verify this.

My email is <email address hidden>.

-- Jim

[Jim Nelson (yorba-jim)]

The duplicate problem is due to these files having the same thumbnail. The ticket I mentioned earlier (http://trac.yorba.org/ticket/2587) addresses this.

I've also ticketed the request to log all files not imported, so that they can be inspected: http://trac.yorba.org/ticket/2593

[john.yarger (john-yarger)]

I just imported my 30,500 photos from f-spot and got a similar response from my import. My assumption was that the import results would be logged to file somewhere so the issues could be researched/resolved. I think the true user requirement here is logging which files failed to import. I had over 400 of my photos reportedly not import correctly, so I don't feel comfortable leaving f-spot until this is resolved. (The significant root cause, I suspect, are broken links resulting from my use of f-spot's "UNSTABLE" ChangePath Tool to move my library to a NAS)

Here is my other quick feedback on my f-spot import and subsequent use of Shotwell:
(1) tag categories (people, events, places) are not retained or create-able within Shotwell
(2) tags are not assignable directly with a right mouse button -- without typing
(3) while the assignment of 5 stars to my f-spot Favorites worked 100%, many non-favorites were assigned 5 stars (I never used f-spots' star rating system)
(4) f-spot's Hidden category is not recognized or supported -- this is an important feature in my family's opinion.
(5) file name is not visible in basic information view; extended information view is a pop up (why?) that

I hope Shotwell will continue to be improved and these feature shortcomings addressed. I suspect that I will wait until Ubuntu 11.04 comes out with an updated Shotwell -- hopefully sporting an improved f-spot import function that handles this concerns.

--John

[Jim Nelson (yorba-jim)]

(1) Tag categories are something we want to have in Shotwell by way of heirarchical tagging: http://trac.yorba.org/ticket/1401 This is a heavily requested feature and something we'd like to get done soon. When we get that implemented we should also update the F-Spot importer to utilize them.

(2) Using the right mouse button to add a tag -- I suspect you mean a submenu in the context menu that lists all tags that can be selected. This is possible, but when the list of tags grows it will become unwieldy. Note that you can assign a tag to a photo by dragging it onto the tag in the sidebar.

(3) It's possible the photos that were marked 5 stars (but were not 5 stars in F-Spot) have rating metadata embedded in them from another source, in which case Shotwell respects that. You can verify this by installing the exiv2 command line tool (if it's not already):

$ sudo apt-get install exiv2

and then running this command on one of the files:

$ exiv2 -pa <filename>

If you see an Xmp.xmp.Rating, Iptc.Application2.Urgency, or Xmp.photoshop.Urgency listed as 5, that would explain it.

(4) The F-Spot importer should convert hidden photos to Shotwell's Rejected rating, which is hidden by default. If you type Ctrl+0, do you still see them? What version of F-Spot are you upgrading from?

(5) Some users in the past complained that the raw filename was not very useful to them (especially for photos coming off a camera as DSC0003.JPG). Shotwell will display the filename in the Basic Information pane if no title has been set, otherwise it displays the title. You can also display the title by choosing View -> Titles.

-- Jim

[john.yarger (john-yarger)]

Jim,
Thanks for taking the time to thoroughly digest and quickly responding to my observations and concerns. And also teaching me the proper terms to use.

(1) It's good to hear that hierarchical tagging is coming soon.

(2) I believe that prior to 0.8.0, f-spot provided this type of right-mouse-button submenu using their hierarchical tagging to make it manageable -- but it appears to have removed now. Your suggest is good and would work even better if we could assign a photo to represent each tag and hierarchical tagging is implemented.

(3) A little more analysis on my part regarding those extra photos assigned 5 star ratings led me to realize an answer to a question I had. How would the importer handle the "modified" copies of files that f-spot stores in the same directories? The non-"Favorite" 5-star photos were always next to cropped-in "Favorite" 5-star photos. The importer appears to simply import both the original photo and the modified photo -- not much different than simply importing the actual photo library directly (since I have tags written to the files). Although I realize this is contrary to Shotwell's philosophy of storing the edits in a db (similar to picasa?), it would be nice if the two photos could be associated with each other like in f-spot. It would be super if a some code would be added to associate them in a way that the "Revert to Original" function to bring up the original photo.

(4) Thanks for cluing me in on the Rejected rating. I finally found "View->Filter Photos". I saw there that Ctrl+9 will show the All + Rejected. And it did. Interestingly, f-spot wrote "Hidden" to the photo file probably as a result of me configuring it to so with tags.

(5) During my import, the filename was not injected as the title for any of my photos. The title field is empty for each.

A couple additional questions/observations:
(6) Are there any plans to log the results of the import functions?

(7) The wiki's UsingShotwell0.7 does not describe how new photos are actually stored in the library after import. Is it by year/month/day of exposure, like f-spot?

(8) Are there plans to create an integrated Shotwell screensaver for gnome, like f-spot, of the favorite/5-star photos?

(9) I just imported my entire photo library on top of the f-spot import. It identified something like 33,000 duplicates and only imported 146 which must have been the broken linked photos Shotwell could not import based on my (slightly corrupted) f-spot db.

My learning Shotwell continues. Thanks for you assistance.

--John

[Jim Nelson (yorba-jim)]

Hi John,

We should probably take this elsewhere, as this bug ticket isn't the proper place to discuss all these different features and requests. Could you post this on our mailing list, where I can respond properly? You can join at http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell

Thanks!

-- Jim

[Ananth P (ananthp)]

(shotwell) '6314 duplicate photos were not imported'
I'd like to know what are those files, so I can erase them to clean up some space. Unfortunately, the filenames were not logged. Should be logged. +1

[Γουργιώτης Γιώργος (aka Gourgi) (gourgi)]

this bug is still present in shotwell 0.9

[wensveen (wensveen)]

A log file containing the full paths of all of the problematic photos would be very useful. For me it's only a handful of duplicates, but I like to keep things tidy.

[Giovanni Mellini (merlos)]

I got this error while moving my pictures to a brand new installation of Precise (Shotwell 0.11.93-0ubuntu2).
I just copied my whole library (11940 pictures/movies, 34,6 GB total) in the Images directory and started Shotwell.
I said to import the Images directory and I got a similar report from Shotwell.

Now I cannot see which file are missing and I cannot import it anymore.
Can anyone help??

[Laura Khalil (loura)]

Giovanni,

If I understand correctly, the issue that you are having is that you cannot see all the files that were marked as duplicates when you imported photos into Shotwell.

We currently have an outstanding feature logged upstream to log all files not imported:

http://redmine.yorba.org/issues/2593

There are many features we would like to include in Shotwell. If you have a Redmine account, you may follow this issue upstream and comment on its progress.

[Giovanni Mellini (merlos)]

Tks Laura.
I finnaly imported all my pictures. I discovered launching shotwell via command line
 SHOTWELL_LOG=1 shotwell
that the images not imported give me the error "Unable to perform MD5 checksum on file". It was a problem with the local copy of these files. I copied it again, import and now is ok.
I think is useful for users to hare a more accurated report of which files are not imported, and a detail of why these files are not imported

Tks
Giovanni