Pentax MakerNote tags advertised with the wrong type, decoding their value raises a ValueError

[Hobson Lane]

Traceback for pyexiv 0.3 below:

  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 189, in _get_value
    self._compute_value()
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 184, in _compute_value
    self._value = self._convert_to_python(self._raw_value)
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 332, in _convert_to_python
    return undefined_to_string(value)
  File "/usr/lib/python2.7/dist-packages/pyexiv2/utils.py", line 148, in undefined_to_string
    return ''.join(map(lambda x: chr(int(x)), undefined.rstrip().split(' ')))
  File "/usr/lib/python2.7/dist-packages/pyexiv2/utils.py", line 148, in <lambda>
    return ''.join(map(lambda x: chr(int(x)), undefined.rstrip().split(' ')))
ValueError: chr() arg not in range(256)

im = pyexiv2.ImageMetadata('IMGP1118sm.JPG')
im.read()
for k in im.exif_keys:
        s = im[k].value;
        if s == 'Unknown':
            s = im[k].raw_value # display raw value for unknown interpretations of the Exif value in exiv2 library
        print "{0}: {1}".format(k,s)

Simplest fix is to mod the integer by 256, "e.g. int(x) % mod 256" before converting to a character and appending to a string. A more robust fix would translate or escape on all non-printable characters before outputing the character string. Suggested patches to follow..

[Olivier Tilloy (osomon)]

A ValueError raised in that case is intended, because the conversion fails.

The real issue is the root cause of this failure: the faulty tag is 'Exif.Pentax.PreviewResolution', and pyexiv2 thinks its type is 'Undefined', as advertised by the documentation (see http://exiv2.org/tags-pentax.html), where in fact it should be 'Short'.
It seems that exiv2 itself knows how to fix the type, because the command line tool returns the correct type:

$ exiv2 -PEkyvt IMGP1118sm.JPG | grep PreviewResolution
Exif.Pentax.PreviewResolution Short 640 480 640x480

This may very well be an issue with the way pyexiv2 queries libexiv2 for the type of a tag.

[Olivier Tilloy (osomon)]

By the way, thanks for the report and the patch Hobson!
As it stands, your patch is merely a workaround and doesn’t fix the actual issue.
I’d rather investigate the cause of the issue and fix it at its source. Of course, patches for this are welcome!

[Hobson Lane (hobs)]

Sounds good. I've since discovered a similar error for the "fractional"
datatype in Nikon exif data. Will look into it some more when I get a
chance.
The patch might be useful as "suspenders" to back up the "belt" in case
manufacturers use nonstandard types or have errors in their compliance with
standards.
On May 19, 2011 1:33 AM, "Olivier Tilloy" <email address hidden> wrote:
> By the way, thanks for the report and the patch Hobson!
> As it stands, your patch is merely a workaround and doesn’t fix the actual
issue.
> I’d rather investigate the cause of the issue and fix it at its source. Of
course, patches for this are welcome!
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/781464
>
> Title:
> Malformed exif tags cause ValueError on Ubuntu Natty Narwhal
>
> Status in pyexiv2, a python binding to exiv2:
> Confirmed
>
> Bug description:
> Traceback for pyexiv 0.3 below:
>
> File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 189, in
_get_value
> self._compute_value()
> File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 184, in
_compute_value
> self._value = self._convert_to_python(self._raw_value)
> File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 332, in
_convert_to_python
> return undefined_to_string(value)
> File "/usr/lib/python2.7/dist-packages/pyexiv2/utils.py", line 148, in
undefined_to_string
> return ''.join(map(lambda x: chr(int(x)), undefined.rstrip().split(' ')))
> File "/usr/lib/python2.7/dist-packages/pyexiv2/utils.py", line 148, in
<lambda>
> return ''.join(map(lambda x: chr(int(x)), undefined.rstrip().split(' ')))
> ValueError: chr() arg not in range(256)
>
>
> im = pyexiv2.ImageMetadata('IMGP1118sm.JPG')
> im.read()
> for k in im.exif_keys:
> s = im[k].value;
> if s == 'Unknown':
> s = im[k].raw_value # display raw value for unknown interpretations of the
Exif value in exiv2 library
> print "{0}: {1}".format(k,s)
>
> Simplest fix is to mod the integer by 256, "e.g. int(x) % mod 256"
> before converting to a character and appending to a string. A more
> robust fix would translate or escape on all non-printable characters
> before outputing the character string. Suggested patches to follow..
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/pyexiv2/+bug/781464/+subscribe

[Olivier Tilloy (osomon)]

In the image attached, accessing the value of the following tags raises a ValueError. All of them have type Undefined according to http://www.exiv2.org/tags-pentax.html. Between parenthesis is the correct type advertised by the exiv2 command-line tool.

<Exif.Pentax.PreviewResolution [Undefined] = 640 480> (Short)
<Exif.Pentax.PreviewLength [Undefined] = 22377> (Long)
<Exif.Pentax.PreviewOffset [Undefined] = 2256> (Long)
<Exif.Pentax.WhiteBallanceMode [Undefined] = 65535> (Short)
<Exif.Pentax.FocalLength [Undefined] = 1800> (Long)
<Exif.Pentax.WhitePoint [Undefined] = 13632 8192 8192 8576> (Short)
<Exif.Pentax.WB_RGGBLevelsDaylight [Undefined] = 13600 8192 8192 8765> (Short)
<Exif.Pentax.WB_RGGBLevelsShade [Undefined] = 16128 8192 8192 6635> (Short)
<Exif.Pentax.WB_RGGBLevelsCloudy [Undefined] = 14560 8192 8192 7782> (Short)
<Exif.Pentax.WB_RGGBLevelsTungsten [Undefined] = 8192 8192 8192 20971> (Short)
<Exif.Pentax.WB_RGGBLevelsFluorescentD [Undefined] = 17376 8192 8192 8847> (Short)
<Exif.Pentax.WB_RGGBLevelsFluorescentN [Undefined] = 14528 8192 8192 10076> (Short)
<Exif.Pentax.WB_RGGBLevelsFluorescentW [Undefined] = 13088 8192 8192 12206> (Short)
<Exif.Pentax.WB_RGGBLevelsFlash [Undefined] = 13632 8192 8192 8601> (Short)
<Exif.Pentax.CameraInfo [Undefined] = 76830 20070527 2 1 4228109> (Long)

[Olivier Tilloy (osomon)]

This bug is in fact very similar to bug #677267, only that it is with EXIF MakerNote tags, not with XMP tags.

Where possible, the type of the tag should be extracted from the existing Exifdatum using datum->typeName() instead of Exiv2::TypeInfo::typeName(exifKey.defaultTypeId()).

[Olivier Tilloy (osomon)]

Fixed at revision 365 of lp:~osomon/pyexiv2/pyexiv2-0.3.

[Hobson Lane (hobs)]

I still get the error on my machine (Natty, Ubu 11.04) for revno 367. I never found the root cause--took the lazy way out with the previously suggested mod 256 patch. After all line148 pyexiv2/utils casts to int before chr, so why not clip or wrap the int at 8 bits (ASCII), in case wayward camera firmware doesn't.

In [7]: run show_bug1.py
Exif.Image.ImageDescription:
Exif.Image.Make: NIKON
Exif.Image.Model: COOLPIX L18
Exif.Image.Orientation: 1
Exif.Image.XResolution: 72
Exif.Image.YResolution: 72
Exif.Image.ResolutionUnit: 2
Exif.Image.Software: GIMP 2.6.11
Exif.Image.DateTime: 2011-05-21 12:10:27
Exif.Image.YCbCrPositioning: 2
Exif.Image.ExifTag: 224
Exif.Photo.ExposureTime: 1/125
Exif.Photo.FNumber: 41/10
Exif.Photo.ExposureProgram: 2
Exif.Photo.ISOSpeedRatings: 200
Exif.Photo.ExifVersion: 0220
Exif.Photo.DateTimeOriginal: 2009-07-27 15:57:06
Exif.Photo.DateTimeDigitized: 2009-07-27 15:57:06
Exif.Photo.ComponentsConfiguration:

Exif.Photo.CompressedBitsPerPixel: 2
Exif.Photo.ExposureBiasValue: 0
Exif.Photo.MaxApertureValue: 3
Exif.Photo.MeteringMode: 5
Exif.Photo.LightSource: 0
Exif.Photo.Flash: 25
Exif.Photo.FocalLength: 13727/1000
�
�
��
.MakerNote: NikonII
�

Z�
g�
o��
w
���
���
�
��
    ��
��
�
COLOR_NORMALAUTO AUTO AF-S NORMAL
�AUTO
@
�NORMAL OFF dd OFF VR-ON NORMAL
Exif.MakerNote.Offset: 738
Exif.MakerNote.ByteOrder: II
Exif.Nikon3.Version: 
Exif.Nikon3.ISOSpeed: [0, 0]
Exif.Nikon3.ColorMode: COLOR_
Exif.Nikon3.Quality: NORMAL
Exif.Nikon3.WhiteBalance: AUTO
Exif.Nikon3.Sharpening: AUTO
Exif.Nikon3.Focus: AF-S
Exif.Nikon3.FlashSetting: NORMAL
Exif.Nikon3.0x000a: 7183/1000
Exif.Nikon3.ISOSelection: AUTO
Exif.Nikon3.DataDump:
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (26, 0))

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)

/media/Win7/Users/Hobs/Documents/Photos/malformed_exif_examples/show_bug1.py in <module>()
      6 im.read()
      7 for k in im.exif_keys:
----> 8 print "{0}: {1}".format(k,im[k].value)
      9
     10

/usr/lib/python2.7/dist-packages/pyexiv2/exif.pyc in _get_value(self)
    188 def _get_value(self):
    189 if self._value_cookie:
--> 190 self._compute_value()
    191 return self._value
    192

/usr/lib/python2.7/dist-packages/pyexiv2/exif.pyc in _compute_value(self)
    183 return
    184
--> 185 self._value = self._convert_to_python(self._raw_value)
    186 self._value_cookie = False
    187

/usr/lib/python2.7/dist-packages/pyexiv2/exif.pyc in _convert_to_python(self, value)
    331 # TODO: guess the encoding and decode accordingly into unicode

    332 # where relevant.

--> 333 return undefined_to_string(value)
    334
    335 raise ExifValueError(value, self.type)

/usr/lib/python2.7/dist-packag...

[Olivier Tilloy (osomon)]

Hobson, the latest trunk works fine for me here with the image you attached ("bora_bora--stone_tiki.JPG").

It looks like you’re not running from trunk but the version installed system-wide. If you want to run from trunk, you’ll have to compile it and install it first. To do that, cd to the top-level directory of the branch, and run the following commands:

    scons
    scons install --user

This will install it in your $HOME directory (in $HOME/.local/lib/python2.7/site-packages/), so it won’t conflict with the system-wide install, and at import time Python will look up for the module there first.

If you’re still seeing the issue, please let me know.

[Hobson Lane (hobs)]

Thanks, Oliver. Had to install some -dev packages to get the scons build to work, but all is well now. Thanks!

    sudo aptitude install libboost-python-dev libexiv2-dev
    scons
    sudo scons install --user

[Olivier Tilloy (osomon)]

Thanks for the feedback Hobson.
Note that next time you need to install the packages needed to build an existing source package, you can use apt-get build-dep:

    sudo apt-get build-dep pyexiv2

This will install all the packages necessary to building pyexiv2.