Date information not imported with jpg files of scanned negatives

[Michael Hendry]

Binary package hint: shotwell

Version 0.72 of Shotwell.

$ apt-cache policy shotwell
shotwell:
  Installed: 0.7.2-1~lucid1
  Candidate: 0.7.2-1~lucid1
  Version table:
 *** 0.7.2-1~lucid1 0
        500 http://ppa.launchpad.net/yorba/ppa/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status
     0.5.0+dfsg-1.1 0
        500 http://gb.archive.ubuntu.com/ubuntu/ lucid/universe Packages

$ lsb_release -rd
Description: Ubuntu 10.04.1 LTS
Release: 10.04

Just imported > 21000 image files, including scans of negatives going back to the 1960s.

Digital camera images organised by date satisfactorily.

Scan date ignored for images scanned from negatives using Nikon Coolscan IV, which makes indexing more difficult.

Date of scan is available through File Manager->Right Click on Filename->Properties->Image

[Michael Hendry (hendry-michael)]

Looking further into this, I find that although the File Manager appears to report the date of the scan, it's actually reporting the date of change of the file - the scanning software has not included the date of the scan in the EXIF metadata associated with the image.

It would still be helpful if images could be sorted by the date the file was created or modified in the absence of EXIF date information, but that would be a Request For Change.

[Adam Dingle (adam-yorba)]

Michael,

this is ticketed upstream at http://trac.yorba.org/ticket/1212 . We do have mixed feelings, however, about using the file time for sorting/grouping when no EXIF date is present, since we think that for many photos this time will not be relevant. We might allow this as a user option.

See also http://trac.yorba.org/ticket/1184 . I don't know whether your scanning software is storing DateTimeDigitized, however.

[Michael Hendry (hendry-michael)]

On Tue, 2010-09-21 at 17:16 +0000, Adam Dingle wrote:
> Michael,
>
> this is ticketed upstream at http://trac.yorba.org/ticket/1212 . We do
> have mixed feelings, however, about using the file time for
> sorting/grouping when no EXIF date is present, since we think that for
> many photos this time will not be relevant. We might allow this as a
> user option.

Many thanks for your prompt response, Adam.

I've been looking at "exiftool" as a means of adjusting the date held in
the metadata in a batch of files, but as you say, the date may not be
particularly relevant to the images themselves.

I have my old negatives filed by the year the film was originally
developed, with a serial number (e.g. 1972-01, 1972-02 etc), and I've
adopted the same convention when scanning, using the number on the
negative as the filename within a subfolder for each film.

The first image on the first film developed in 1976 is stored in
directory ./1973-01 and entitled IMG0001.jpg, the second IMG0002.jpg,
etc.

Ideally, I'd like Shotwell to display images in chronological order, so
I suppose I could use exiftool to say that IMAGE0001.jpg was taken on
1st Jan 1973 at 00:01, IMAGE0002.jpg at 00:02 and so on. It should be
obvious to anyone looking at these dates and times later that they are
arbitrary.

If I could achieve this, would Shotwell do this for me? Would I have to
re-import all my images for the changes to the EXIF data to be picked
up?

>
> See also http://trac.yorba.org/ticket/1184 . I don't know whether your
> scanning software is storing DateTimeDigitized, however.

No DateTimeDigitized is reported by "exiftool -a IMG0001.jpg".

There is a "Modify Date" which is slightly earlier than the "File
Modification Date/Time", and I think reflects the scanner's view of the
time it finished the job as opposed to the operating system's datestamp.

>
> ** Bug watch added: trac.yorba.org/ #1212
> http://trac.yorba.org/ticket/1212

>
> ** Bug watch added: trac.yorba.org/ #1184
> http://trac.yorba.org/ticket/1184
>

Regards,

Michael

[Jim Nelson (yorba-jim)]

Michael,

The current version of Shotwell (0.7.x) will not notice the change to the EXIF data and re-sort your photos. That is planned for the next release, and code to do that is in trunk today. If you're curious, check out our project page at http://www.yorba.org/shotwell/install/#source for building the latest. Otherwise, you would need to re-import your photos after updating their metadata.

The Modify Date that exiftool reports is actually EXIF's DateTime tag -- yet another quasi-ambiguous date field in the EXIF spec. Usually DateTime is specified along with DateTimeOriginal and/or DateTimeDigitzed, so that might be another field we should fall back on. I'll update our ticket to consider this.

Could you install the exiv2 command line tool (if it's not already installed) and run this command:

$ exiv2 -pa image.jpg > exiv2.txt

and attach the exiv2.txt file to this ticket? That would give me a better idea of what metadata is stored in your scanned files. Thanks!

[Michael Hendry (hendry-michael)]

On Tue, 2010-09-21 at 18:39 +0000, Jim Nelson wrote:
> Michael,
>
> The current version of Shotwell (0.7.x) will not notice the change to
> the EXIF data and re-sort your photos.

That's what I suspected.

>
> That is planned for the next
> release, and code to do that is in trunk today.

There's obviously going to be a time penalty if every file needs to be
checked - remember I've got > 20,000 - it may need to be a configurable
option, or perhaps a "Rescan" option in the "File" menu?

> If you're curious,
> check out our project page at
> http://www.yorba.org/shotwell/install/#source for building the latest.
> Otherwise, you would need to re-import your photos after updating their
> metadata.
>
> The Modify Date that exiftool reports is actually EXIF's DateTime tag --
> yet another quasi-ambiguous date field in the EXIF spec. Usually
> DateTime is specified along with DateTimeOriginal and/or
> DateTimeDigitzed, so that might be another field we should fall back on.
> I'll update our ticket to consider this.
>
> Could you install the exiv2 command line tool (if it's not already
> installed) and run this command:
>
> $ exiv2 -pa image.jpg > exiv2.txt
>
> and attach the exiv2.txt file to this ticket? That would give me a
> better idea of what metadata is stored in your scanned files. Thanks!
>

I'm not sure what you mean by "...attach... to this ticket", so I've
attached it to this e-mail, and I've also attached the output of
"exiftool -a IMG001.jpg", which applies different names to the
properties, just to make life easy for us.

I think the date of the scan itself is given by:

Exif.Image.DateTime Ascii 20 2002.04.07 09.47.26

and

Modify Date : 2002.04.07 09.47.26

respectively by exiv2 and exiftool, while the OS's timestamp as reported
by exiftool is:

File Modification Date/Time : 2002:04:07 09:50:18+01:00

Michael

PS I see that exiftool says that the Primary Platform is "Apple Computer
Inc.", which is odd, because I have never owned an Apple computer, and
the original scan was definitely made on a PC.

[Michael Hendry (hendry-michael)]

As a new Linux user, I've been dipping my toes in the water with various applications, trying to find the best way to make the transition from Windows.

My first attempt in the photographic side of things was with f-spot, which was distributed as part of the standard Lucid Lynx distribution, and which I'd used to import a few images from my camera.

Something went wrong with f-spot last week, making it crash every time I pressed the import button, and after casting around for a solution for a while I eventually decided to drop it and use shotwell.

Oddly enough, f-spot miraculously recovered yesteray, after an Ubuntu update, so I tried importing some scanned files into f-spot, which does pick up the date on which an image was scanned.

However, any modification to a file changes the date that f-spot uses for indexing, which means that a film which has been scanned and processed in a single session will be scattered all over the timeline when the images are edited. Ideally, I'd like to keep all images and their edited versions together.

Clearly I'm going to have to study EXIF carefully before making any large-scale changes to the metadata stored with my images!

I'm planning to stick with shotwell!

[Jim Nelson (yorba-jim)]

Hi Michael,

Thanks for the text files. This verifies my suspicions, that the scanner is using the plain DateTime EXIF tag and not DateTimeDigitized or DateTimeOriginal. If we use a fallback scheme (first looking for DateTimeOriginal, then DateTimeDigitized, then DateTime), this should solve your problem. We're still on the fence about how, or if, to use the file's modification time, but there is some rationale to that.

No idea about the Apple tur-- ah, *tag* -- in your EXIF data. Perhaps Nikon doesn't expect people to use its hardware on anything but a Mac.

Glad to hear that Shotwell is meeting your needs, even if we're still catching up on this point. Hopefully we'll have this fixed for you shortly.

-- Jim

[Michael Hendry (hendry-michael)]

I've been reading about the EXIF "standard", and it appears not quite to match up to that description!

One problem is that the tags are identified by number within the image file, and different applications attach different descriptions to these numbers.

What seems clear is that there are three distinct date and time tags (with fractions of seconds in separate tags to go with them): taking one set of names, DateTimeOriginal is intended to denote the date of creation of the original object, DateTimeDigitized is for literally that, the date this digital copy was first made of the object, and DateTime is the date the file in which the image is contained was last changed.

Shotwell is built around the idea of "Events", grouping images together by the date on which they (appear to have) occurred, so it's logical that DateTimeOriginal should take precedence. If this is absent, then DateTimeDigitized should be used, as this denotes an "event" in this image's history, and would tend to group images together in a chronological sequence (as a length of 35mm still film is scanned in).

My scanner uses DateTime only, and it would be helpful if this could be picked up in the absence of the other two, and finally there's the operating system's creation date for the file - which is all you'll get (for example) in a BMP file.

When I first got my scanner I used it to save the files in BMP format at maximum resolution, reasoning that this would include all the information the scanner could get from the image, but as my primary interest at that stage was the creation of some sort of index of all my old negatives I cut back the resolution and reverted to JPG to get the job done in a reasonable time without taking over all my hard disc space. Promising images can be re-scanned in a higher resolution.

Is there any particular reason why you don't support BMP? I've started converting them to JPG so as to be able to get Shotwell to see them, but this is a tiresome chore, and takes up more disc space. (Off-topic, I know...).

[Jim Nelson (yorba-jim)]

We've not received much interest in supporting BMP, which is why we've not implemented it. We do have a ticket to add support for it (as well as GIF) here: http://trac.yorba.org/ticket/2154 If you want to continue discussing that feature, that would be a good place to do it.

I now definitely think we should use the fallback scheme I described (and which you agree with). I've upped the priority for that ticket to high, and hopefully we can get this in for 0.8.

-- Jim

[Michael Hendry (hendry-michael)]

Thanks, Jim.

Once the fallback scheme has been implemented, will I have to remove all affected images from the image library and then re-import them? If that's the case, I'll lose all the work I've already done in tagging the files. As the files themselves won't have changed, they'll presumably appear to be duplicates and they'll be rejected by Import.

[Jim Nelson (yorba-jim)]

I added a note to our ticket detailing an upgrade scheme that should solve the problem you're describing (thanks for pointing it out). It would prevent you from having to re-import the photos. The only open question is whether or not to create new events for these photos once their DateTime is determined. My inclination right now is to do so, but I want to think it through further.

[Eric Gregory (eric-yorba)]

We've decided that for Shotwell 0.8, we're only going to read the extended timestamp EXIF information from new photos, since rescanning photos that were already imported and updating the database is a fairly large change.

However, I've filed a new ticket for the rescan/database update:
http://trac.yorba.org/ticket/2910

[Omer Akram (om26er)]

shotwell 0.8.1 is now in Natty

[Michael Hendry (hendry-michael)]

On Tue, 2011-01-18 at 20:05 +0000, Omer Akram wrote:
> shotwell 0.8.1 is now in Natty
>
> ** Changed in: shotwell (Ubuntu)
> Status: Triaged => Fix Released
>

Thanks Omer.

I'm using 0.7.2 with Lucid Lynx - do I have to upgrade to a later
version of Ubuntu to use 0.8.1?

Michael

[Adam Dingle (adam-yorba)]

Michael,

Shotwell 0.8.1 will run on Lucid, but neither the Ubuntu repositories nor the Yorba PPA contain Shotwell 0.8.1 for Lucid - you need to build it yourself. See the instructions on the Yorba web site, and feel free to ask on the Shotwell mailing list if you're having trouble building.