xmp sidecar files not supported

Bug #910003 reported by Damon Lynch
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
pyexiv2
New
Wishlist
Unassigned

Bug Description

Currently exiv2 can write to XMP sidecar files, extracting and converting metadata from the image.

exiv2 can also set new values and write them out to XMP sidecar files, irrespective of any data in the image itself, e.g. with the command -M"set Xmp.iptc.CreatorContactInfo/Iptc4xmpCore:CiAdrCity Boston" (the exiv2 documentation is quite confusing for instances like this, but this particular example is correct).

exiv2 has limited read capacity from XMP sidecar files, in that it can immediately insert directly from a sidecar into an image.

In contrast to exiv2, pyexiv2 lacks the functionality to write XMP sidecar files. This is a problem especially when creating XMP metadata for images formats that are dangerous or foolish to write to -- such as non-DNG RAW files -- where new metadata really does belong in an XMP sidecar.

Moreover, when reading XMP data from an image, the XMP sidecar is totally ignored. This mimics exiv2, but it would be more useful to have the option for pyexiv2 to assume that metadata in the most recent file of the XMP sidecar - image pair is higher priority. This would mimic the behavior of Adobe Bridge and Bibble, for instance. (The discussion in the subsection "PhotoMechanic issues" is useful: http://www.dpbestflow.org/file-lifecycle/ingestion#ingestion).

Versions:
pyexiv 0.3.2
exiv2 0.21.1

Damon Lynch (dlynch3)
description: updated
Olivier Tilloy (osomon)
Changed in pyexiv2:
importance: Undecided → Wishlist
Revision history for this message
Damon Lynch (dlynch3) wrote :

I have been attempting my own modest implementation of this request. After actually coding it, I ran into a serious problem: writing the XMP sidecar file fails to modify any existing values. This might be purely because of a problem in exiv2: http://dev.exiv2.org/boards/3/topics/1052

Revision history for this message
dmoore (damien-moore) wrote :

Is there any progress on this?

I tried a small script that read in an XMP file, changed a value and wrote back to the file, but it appears to still not be working.

>Moreover, when reading XMP data from an image, the XMP sidecar is totally ignored.
>This mimics exiv2, but it would be more useful to have the option for pyexiv2 to assume
>that metadata in the most recent file of the XMP sidecar - image pair is higher priority.

I don't know if that is the best approach. First, it appears that there isn't a standard naming scheme for sidecars: some use image.xmp (totally bonkers) others image.jpg.xmp (yay!). Second, in the few sidecars I have on my system, it appears that the apps write only "extra" data to the sidecar, so you would need to load both the image and the sidecar data to have a complete record. Third, to implement this feature would require some mappings between Exif/Iptc keys to their sidecar representations, which don't look like they are quite the same because the Xmp schema for say Exif is different (names/types). Compare:

http://www.exiv2.org/tags.html

http://www.exiv2.org/tags-xmp-exif.html

Revision history for this message
Damon Lynch (dlynch3) wrote : Re: [Bug 910003] Re: xmp sidecar files not supported

I strongly disagree with the previous bug poster's comments. It is
preferable to follow recognized industry standards, rather than make
them up as we go along. There are widely recognized standards for how
the XMP files should be named and what should be in them.

Adobe has developed a comprehensive XMP spec. There are important
technical specifications here too:
http://www.metadataworkinggroup.org/ The latter details how to resolve
metadata conflicts.

Revision history for this message
dmoore (damien-moore) wrote :
Download full text (3.8 KiB)

I guess I wasn't very clear. I'm all for standards. I wrote my own photo manager app in part because I was very unhappy with the handling of metadata in the available open source photo managers. (The handling in my app sucks too, but it sucks in a way I understand, and I'm trying to move in the right direction.)

My point was that this isn't just a handful of small changes to make the API a bit friendlier, as my last comment and your last comment illustrate. There is a lot of work involved to get this right. With this in mind, I would suggest:

(a) Create a separate python lib that builds on pyexiv2.
(b) Lobby for these features in exiv2

The virtue of both approaches is that they wouldn't hold up the release of new versions of pyexiv2, and hence, make it easier to keep up with the changes in exiv2. The advantage of (a) over (b) is that it would further the goal of interoperability as more apps, i.e. beyond those written in python, would be able to rely on a common set of features. The advantage of (b) over (a) is that it it is probably much easier to write this sort of code in python than C++.

Back to the subject of this bug. I played around with this some more and it looks like writing metadata to sidecars that already exist does indeed work, at least for my test case. Interestingly, when the data is written the lib attempts to convert iptc or exif tags to the equivalent xmp tag (assuming the xmp tag hasn't already been set). I assume this is exiv2 behavior and not something added to pyexiv2(?). However, the writer doesn't sync up the equivalent tags if both are present.

Here's sample code & output (note some unexpected behavior):

>>> i = pyexiv2.ImageMetadata("FLoral.jpg.xmp")
>>> i.read()
>>> i['Xmp.dc.subject'].value
[u'key', u'word']
>>> i["Iptc.Application2.Keywords"].value
['key', 'word']
>>> i['Xmp.dc.subject']=['a','b']
>>> i["Iptc.Application2.Keywords"].value
['key', 'word']
>>> i["Iptc.Application2.Keywords"]=['c','d']
>>> i["Iptc.Application2.Keywords"].value
['c', 'd']
>>> i['Xmp.dc.subject'].value
['a', 'b']
>>> i.write()
>>> i['Xmp.dc.subject'].value
['a', 'b']
>>> i["Iptc.Application2.Keywords"].value
['c', 'd']
>>> i.read()
>>> i["Iptc.Application2.Keywords"].value
['c', 'd']
>>> i['Xmp.dc.subject'].value
['a', 'b']
>>> j = pyexiv2.ImageMetadata("FLoral.jpg.xmp")
>>> j.read()
>>> j['Xmp.dc.subject'].value
[u'c', u'd']
>>> j["Iptc.Application2.Keywords"].value
['c', 'd']

The source xmp file is below. You can see the junk written into it by darktable and the lack of exif data etc that I gather should be present if this program was following the standards:

<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""
    xmlns:xmp="http://ns.adobe.com/xap/1.0/"
    xmlns:darktable="http://darktable.sf.net/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
   xmp:Rating="1"
   darktable:xmp_version="1"
   darktable:raw_params="-16711648"
   photoshop:Credit="Lorenzo">
   <darktable:colorlabels>
    <rdf:Seq/>
   </darktable:...

Read more...

Revision history for this message
dmoore (damien-moore) wrote :

Here's some workaround code to create a new sidecar from the metadata in an existing image. Obviously, this code can be improved upon.

import pyexiv2

sidecar_stub='''<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
 </rdf:RDF>
</x:xmpmeta>'''

src='/path/to/an/image.jpg'
dest=src+'.xmp'

im = pyexiv2.ImageMetadata(src)
im.read()

f = open(dest,'wb')
f.write(sidecar_stub)
f.close()

sidecar = pyexiv2.ImageMetadata(dest)
sidecar.read()
im.copy(sidecar,comment=False)
sidecar.write()

Revision history for this message
dmoore (damien-moore) wrote :

Below is a workaround for not being able to replace Xmp keys that are already present. It looks like exiv2's "smart" tag conversion has a bug that affects sidecars as Damon noted above. The workaround copies the original sidecar to a new temporary sidecar file then replaces the original file with the temporary one. The secret is to only copy the xmp tags. I wanted to leave a note on Damon's forum post at Exiv2, but haven't been able to successfully create an account. This is worth following up on.

import pyexiv2
import tempfile, os, os.path

filename = '/name/of/sidecar.xmp'

meta_in = pyexiv2.ImageMetadata(filename)
meta_in.read()

sidecar_stub='''<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
 </rdf:RDF>
</x:xmpmeta>'''

tdir,tname=os.path.split(filename)
h,tfilename=tempfile.mkstemp('',tname,tdir)

f=open(tfilename,'wb')
f.write(sidecar_stub)
f.close()
meta_out = pyexiv2.ImageMetadata(tfilename)
meta_out.read()

meta_in['Xmp.dc.title']='A new title 2'
meta_in.copy(meta_out,exif=False, iptc=False, xmp=True,comment=False)
meta_out.write()
os.rename(tfilename,filename)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.